motorhead/qemu
1
0
Fork 0
mirror of https://github.com/Motorhead1991/qemu.git synced 2025-07-29 05:13:54 -06:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs/devel: Split testing docs from the build docs and move to separate folder

Browse source 
Building and testing are two separate topics, so let's split the testing
into a separate category and move the related files into a separate folder.

Message-ID: <20240830133841.142644-42-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This commit is contained in:
Thomas Huth 2024-08-30 15:38:35 +02:00
parent 6d62722ebd
commit ff41da5030
12 changed files with 19 additions and 10 deletions
Download patch file Download diff file Expand all files Collapse all files

155
docs/devel/testing/acpi-bits.rst Normal file
View file

@ -0,0 +1,155 @@
==================================
ACPI/SMBIOS testing using biosbits
==================================
************
Introduction
************
Biosbits is a software written by Josh Triplett that can be downloaded
from https://biosbits.org/. The github codebase can be found
`here <https://github.com/biosbits/bits/tree/master>`__. It is a software that
executes the bios components such as acpi and smbios tables directly through
acpica bios interpreter (a freely available C based library written by Intel,
downloadable from https://acpica.org/ and is included with biosbits) without an
operating system getting involved in between. Bios-bits has python integration
with grub so actual routines that executes bios components can be written in
python instead of bash-ish (grub's native scripting language).
There are several advantages to directly testing the bios in a real physical
machine or in a VM as opposed to indirectly discovering bios issues through the
operating system (the OS). Operating systems tend to bypass bios problems and
hide them from the end user. We have more control of what we wanted to test and
how by being as close to the bios on a running system as possible without a
complicated software component such as an operating system coming in between.
Another issue is that we cannot exercise bios components such as ACPI and
SMBIOS without being in the highest hardware privilege level, ring 0 for
example in case of x86. Since the OS executes from ring 0 whereas normal user
land software resides in unprivileged ring 3, operating system must be modified
in order to write our test routines that exercise and test the bios. This is
not possible in all cases. Lastly, test frameworks and routines are preferably
written using a high level scripting language such as python. OSes and
OS modules are generally written using low level languages such as C and
low level assembly machine language. Writing test routines in a low level
language makes things more cumbersome. These and other reasons makes using
bios-bits very attractive for testing bioses. More details on the inspiration
for developing biosbits and its real life uses can be found in [#a]_ and [#b]_.
For QEMU, we maintain a fork of bios bits in gitlab along with all the
dependent submodules `here <https://gitlab.com/qemu-project/biosbits-bits>`__.
This fork contains numerous fixes, a newer acpica and changes specific to
running these functional QEMU tests using bits. The author of this document
is the sole maintainer of the QEMU fork of bios bits repository. For more
information, please see author's `FOSDEM talk on this bios-bits based test
framework <https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/>`__.
*********************************
Description of the test framework
*********************************
Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU
functional test that drives all this.
A brief description of the various test files follows.
Under ``tests/functional/`` as the root we have:
::
├── acpi-bits
│ ├── bits-config
│ │ └── bits-cfg.txt
│ ├── bits-tests
│ ├── smbios.py2
│ ├── testacpi.py2
│ └── testcpuid.py2
├── test_acpi_bits.py
* ``tests/functional``:
``test_acpi_bits.py``:
This is the main python functional test script that generates a
biosbits iso. It then spawns a QEMU VM with it, collects the log and reports
test failures. This is the script one would be interested in if they wanted
to add or change some component of the log parsing, add a new command line
to alter how QEMU is spawned etc. Test writers typically would not need to
modify this script unless they wanted to enhance or change the log parsing
for their tests. In order to enable debugging, you can set **V=1**
environment variable. This enables verbose mode for the test and also dumps
the entire log from bios bits and more information in case failure happens.
You can also set **BITS_DEBUG=1** to turn on debug mode. It will enable
verbose logs and also retain the temporary work directory the test used for
you to inspect and run the specific commands manually.
In order to run this test, please perform the following steps from the QEMU
build directory (assuming that the sources are in ".."):
::
$ export PYTHONPATH=../python:../tests/functional
$ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64
$ python3 ../tests/functional/test_acpi_bits.py
The above will run all acpi-bits functional tests (producing output in
tap format).
You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/
for more information about the run or in order to diagnoze issues.
If you pass V=1 in the environment, more diagnostic logs will be put into
the test log.
* ``tests/functional/acpi-bits/bits-config``:
This location contains biosbits configuration files that determine how the
software runs the tests.
``bits-config.txt``:
This is the biosbits config file that determines what tests
or actions are performed by bits. The description of the config options are
provided in the file itself.
* ``tests/functional/acpi-bits/bits-tests``:
This directory contains biosbits python based tests that are run from within
the biosbits environment in the spawned VM. New additions of test cases can
be made in the appropriate test file. For example, new acpi tests can go
into testacpi.py2 and one would call testsuite.add_test() to register the new
test so that it gets executed as a part of the ACPI tests.
It might be occasionally necessary to disable some subtests or add a new
test that belongs to a test suite not already present in this directory. To
do this, please clone the bits source from
https://gitlab.com/qemu-project/biosbits-bits/-/tree/qemu-bits.
Note that this is the "qemu-bits" branch and not the "bits" branch of the
repository. "qemu-bits" is the branch where we have made all the QEMU
specific enhancements and we must use the source from this branch only.
Copy the test suite/script that needs modification (addition of new tests
or disabling them) from python directory into this directory. For
example, in order to change cpuid related tests, copy the following
file into this directory and rename it with .py2 extension:
https://gitlab.com/qemu-project/biosbits-bits/-/blob/qemu-bits/python/testcpuid.py
Then make your additions and changes here. Therefore, the steps are:
(a) Copy unmodified test script to this directory from bits source.
(b) Add a SPDX license header.
(c) Perform modifications to the test.
Commits (a), (b) and (c) preferably should go under separate commits so that
the original test script and the changes we have made are separated and
clear. (a) and (b) can sometimes be combined into a single step.
The test framework will then use your modified test script to run the test.
No further changes would be needed. Please check the logs to make sure that
appropriate changes have taken effect.
The tests have an extension .py2 in order to indicate that:
(a) They are python2.7 based scripts and not python 3 scripts.
(b) They are run from within the bios bits VM and is not subjected to QEMU
build/test python script maintenance and dependency resolutions.
(c) They need not be loaded by the test framework by accident when running
tests.
Author: Ani Sinha <anisinha@redhat.com>
References:
-----------
.. [#a] https://blog.linuxplumbersconf.org/2011/ocw/system/presentations/867/original/bits.pdf
.. [#b] https://www.youtube.com/watch?v=36QIepyUuhg
.. [#c] https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/

121
docs/devel/testing/ci-definitions.rst.inc Normal file
View file

@ -0,0 +1,121 @@
Definition of terms
===================
This section defines the terms used in this document and correlates them with
what is currently used on QEMU.
Automated tests
---------------
An automated test is written on a test framework using its generic test
functions/classes. The test framework can run the tests and report their
success or failure [1]_.
An automated test has essentially three parts:
1. The test initialization of the parameters, where the expected parameters,
like inputs and expected results, are set up;
2. The call to the code that should be tested;
3. An assertion, comparing the result from the previous call with the expected
result set during the initialization of the parameters. If the result
matches the expected result, the test has been successful; otherwise, it has
failed.
Unit testing
------------
A unit test is responsible for exercising individual software components as a
unit, like interfaces, data structures, and functionality, uncovering errors
within the boundaries of a component. The verification effort is in the
smallest software unit and focuses on the internal processing logic and data
structures. A test case of unit tests should be designed to uncover errors due
to erroneous computations, incorrect comparisons, or improper control flow [2]_.
On QEMU, unit testing is represented by the 'check-unit' target from 'make'.
Functional testing
------------------
A functional test focuses on the functional requirement of the software.
Deriving sets of input conditions, the functional tests should fully exercise
all the functional requirements for a program. Functional testing is
complementary to other testing techniques, attempting to find errors like
incorrect or missing functions, interface errors, behavior errors, and
initialization and termination errors [3]_.
On QEMU, functional testing is represented by the 'check-qtest' target from
'make'.
System testing
--------------
System tests ensure all application elements mesh properly while the overall
functionality and performance are achieved [4]_. Some or all system components
are integrated to create a complete system to be tested as a whole. System
testing ensures that components are compatible, interact correctly, and
transfer the right data at the right time across their interfaces. As system
testing focuses on interactions, use case-based testing is a practical approach
to system testing [5]_. Note that, in some cases, system testing may require
interaction with third-party software, like operating system images, databases,
networks, and so on.
On QEMU, system testing is represented by the 'check-avocado' target from
'make'.
Flaky tests
-----------
A flaky test is defined as a test that exhibits both a passing and a failing
result with the same code on different runs. Some usual reasons for an
intermittent/flaky test are async wait, concurrency, and test order dependency
[6]_.
Gating
------
A gate restricts the move of code from one stage to another on a
test/deployment pipeline. The step move is granted with approval. The approval
can be a manual intervention or a set of tests succeeding [7]_.
On QEMU, the gating process happens during the pull request. The approval is
done by the project leader running its own set of tests. The pull request gets
merged when the tests succeed.
Continuous Integration (CI)
---------------------------
Continuous integration (CI) requires the builds of the entire application and
the execution of a comprehensive set of automated tests every time there is a
need to commit any set of changes [8]_. The automated tests can be composed of
the unit, functional, system, and other tests.
Keynotes about continuous integration (CI) [9]_:
1. System tests may depend on external software (operating system images,
firmware, database, network).
2. It may take a long time to build and test. It may be impractical to build
the system being developed several times per day.
3. If the development platform is different from the target platform, it may
not be possible to run system tests in the developer’s private workspace.
There may be differences in hardware, operating system, or installed
software. Therefore, more time is required for testing the system.
References
----------
.. [1] Sommerville, Ian (2016). Software Engineering. p. 233.
.. [2] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. p. 48, 376, 378, 381.
.. [3] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. p. 388.
.. [4] Pressman, Roger S. & Maxim, Bruce R. (2020). Software Engineering,
A Practitioner’s Approach. Software Engineering, p. 377.
.. [5] Sommerville, Ian (2016). Software Engineering. p. 59, 232, 240.
.. [6] Luo, Qingzhou, et al. An empirical analysis of flaky tests.
Proceedings of the 22nd ACM SIGSOFT International Symposium on
Foundations of Software Engineering. 2014.
.. [7] Humble, Jez & Farley, David (2010). Continuous Delivery:
Reliable Software Releases Through Build, Test, and Deployment, p. 122.
.. [8] Humble, Jez & Farley, David (2010). Continuous Delivery:
Reliable Software Releases Through Build, Test, and Deployment, p. 55.
.. [9] Sommerville, Ian (2016). Software Engineering. p. 743.

190
docs/devel/testing/ci-jobs.rst.inc Normal file
View file

@ -0,0 +1,190 @@
.. _ci_var:
Custom CI/CD variables
======================
QEMU CI pipelines can be tuned by setting some CI environment variables.
Set variable globally in the user's CI namespace
------------------------------------------------
Variables can be set globally in the user's CI namespace setting.
For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/ci/variables/#add-a-cicd-variable-to-a-project
Set variable manually when pushing a branch or tag to the user's repository
---------------------------------------------------------------------------
Variables can be set manually when pushing a branch or tag, using
git-push command line arguments.
Example setting the QEMU_CI_EXAMPLE_VAR variable:
.. code::
git push -o ci.variable="QEMU_CI_EXAMPLE_VAR=value" myrepo mybranch
For further information about how to set these variables, please refer to::
https://docs.gitlab.com/ee/user/project/push_options.html#push-options-for-gitlab-cicd
Setting aliases in your git config
----------------------------------
You can use aliases to make it easier to push branches with different
CI configurations. For example define an alias for triggering CI:
.. code::
git config --local alias.push-ci "push -o ci.variable=QEMU_CI=1"
git config --local alias.push-ci-now "push -o ci.variable=QEMU_CI=2"
Which lets you run:
.. code::
git push-ci
to create the pipeline, or:
.. code::
git push-ci-now
to create and run the pipeline
Variable naming and grouping
----------------------------
The variables used by QEMU's CI configuration are grouped together
in a handful of namespaces
* QEMU_JOB_nnnn - variables to be defined in individual jobs
or templates, to influence the shared rules defined in the
.base_job_template.
* QEMU_CI_nnn - variables to be set by contributors in their
repository CI settings, or as git push variables, to influence
which jobs get run in a pipeline
* QEMU_CI_CONTAINER_TAG - the tag used to publish containers
in stage 1, for use by build jobs in stage 2. Defaults to
'latest', but if running pipelines for different branches
concurrently, it should be overridden per pipeline.
* QEMU_CI_UPSTREAM - gitlab namespace that is considered to be
the 'upstream'. This defaults to 'qemu-project'. Contributors
may choose to override this if they are modifying rules in
base.yml and need to validate how they will operate when in
an upstream context, as opposed to their fork context.
* nnn - other misc variables not falling into the above
categories, or using different names for historical reasons
and not yet converted.
Maintainer controlled job variables
-----------------------------------
The following variables may be set when defining a job in the
CI configuration file.
QEMU_JOB_CIRRUS
~~~~~~~~~~~~~~~
The job makes use of Cirrus CI infrastructure, requiring the
configuration setup for cirrus-run to be present in the repository
QEMU_JOB_OPTIONAL
~~~~~~~~~~~~~~~~~
The job is expected to be successful in general, but is not run
by default due to need to conserve limited CI resources. It is
available to be started manually by the contributor in the CI
pipelines UI.
QEMU_JOB_ONLY_FORKS
~~~~~~~~~~~~~~~~~~~
The job results are only of interest to contributors prior to
submitting code. They are not required as part of the gating
CI pipeline.
QEMU_JOB_SKIPPED
~~~~~~~~~~~~~~~~
The job is not reliably successful in general, so is not
currently suitable to be run by default. Ideally this should
be a temporary marker until the problems can be addressed, or
the job permanently removed.
QEMU_JOB_PUBLISH
~~~~~~~~~~~~~~~~
The job is for publishing content after a branch has been
merged into the upstream default branch.
QEMU_JOB_AVOCADO
~~~~~~~~~~~~~~~~
The job runs the Avocado integration test suite
Contributor controlled runtime variables
----------------------------------------
The following variables may be set by contributors to control
job execution
QEMU_CI
~~~~~~~
By default, no pipelines will be created on contributor forks
in order to preserve CI credits
Set this variable to 1 to create the pipelines, but leave all
the jobs to be manually started from the UI
Set this variable to 2 to create the pipelines and run all
the jobs immediately, as was the historical behaviour
QEMU_CI_AVOCADO_TESTING
~~~~~~~~~~~~~~~~~~~~~~~
By default, tests using the Avocado framework are not run automatically in
the pipelines (because multiple artifacts have to be downloaded, and if
these artifacts are not already cached, downloading them make the jobs
reach the timeout limit). Set this variable to have the tests using the
Avocado framework run automatically.
Other misc variables
--------------------
These variables are primarily to control execution of jobs on
private runners
AARCH64_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an aarch64 host that can be used as a gitlab-CI
runner, you can set this variable to enable the tests that require this
kind of host. The runner should be tagged with "aarch64".
AARCH32_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an armhf host or an arch64 host that can run
aarch32 EL0 code to be used as a gitlab-CI runner, you can set this
variable to enable the tests that require this kind of host. The
runner should be tagged with "aarch32".
S390X_RUNNER_AVAILABLE
~~~~~~~~~~~~~~~~~~~~~~
If you've got access to an IBM Z host that can be used as a gitlab-CI
runner, you can set this variable to enable the tests that require this
kind of host. The runner should be tagged with "s390x".
CCACHE_DISABLE
~~~~~~~~~~~~~~
The jobs are configured to use "ccache" by default since this typically
reduces compilation time, at the cost of increased storage. If the
use of "ccache" is suspected to be hurting the overall job execution
time, setting the "CCACHE_DISABLE=1" env variable to disable it.

116
docs/devel/testing/ci-runners.rst.inc Normal file
View file

@ -0,0 +1,116 @@
Jobs on Custom Runners
======================
Besides the jobs run under the various CI systems listed before, there
are a number additional jobs that will run before an actual merge.
These use the same GitLab CI's service/framework already used for all
other GitLab based CI jobs, but rely on additional systems, not the
ones provided by GitLab as "shared runners".
The architecture of GitLab's CI service allows different machines to
be set up with GitLab's "agent", called gitlab-runner, which will take
care of running jobs created by events such as a push to a branch.
Here, the combination of a machine, properly configured with GitLab's
gitlab-runner, is called a "custom runner".
The GitLab CI jobs definition for the custom runners are located under::
.gitlab-ci.d/custom-runners.yml
Custom runners entail custom machines. To see a list of the machines
currently deployed in the QEMU GitLab CI and their maintainers, please
refer to the QEMU `wiki <https://wiki.qemu.org/AdminContacts>`__.
Machine Setup Howto
-------------------
For all Linux based systems, the setup can be mostly automated by the
execution of two Ansible playbooks. Create an ``inventory`` file
under ``scripts/ci/setup``, such as this::
fully.qualified.domain
other.machine.hostname
You may need to set some variables in the inventory file itself. One
very common need is to tell Ansible to use a Python 3 interpreter on
those hosts. This would look like::
fully.qualified.domain ansible_python_interpreter=/usr/bin/python3
other.machine.hostname ansible_python_interpreter=/usr/bin/python3
Build environment
~~~~~~~~~~~~~~~~~
The ``scripts/ci/setup/$DISTRO/build-environment.yml`` Ansible
playbook will set up machines with the environment needed to perform
builds and run QEMU tests. This playbook consists on the installation
of various required packages (and a general package update while at
it).
The minimum required version of Ansible successfully tested in this
playbook is 2.8.0 (a version check is embedded within the playbook
itself). To run the playbook, execute::
cd scripts/ci/setup
ansible-playbook -i inventory $DISTRO/build-environment.yml
Please note that most of the tasks in the playbook require superuser
privileges, such as those from the ``root`` account or those obtained
by ``sudo``. If necessary, please refer to ``ansible-playbook``
options such as ``--become``, ``--become-method``, ``--become-user``
and ``--ask-become-pass``.
gitlab-runner setup and registration
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The gitlab-runner agent needs to be installed on each machine that
will run jobs. The association between a machine and a GitLab project
happens with a registration token. To find the registration token for
your repository/project, navigate on GitLab's web UI to:
* Settings (the gears-like icon at the bottom of the left hand side
vertical toolbar), then
* CI/CD, then
* Runners, and click on the "Expand" button, then
* Under "Set up a specific Runner manually", look for the value under
"And this registration token:"
Copy the ``scripts/ci/setup/vars.yml.template`` file to
``scripts/ci/setup/vars.yml``. Then, set the
``gitlab_runner_registration_token`` variable to the value obtained
earlier.
To run the playbook, execute::
cd scripts/ci/setup
ansible-playbook -i inventory gitlab-runner.yml
Following the registration, it's necessary to configure the runner tags,
and optionally other configurations on the GitLab UI. Navigate to:
* Settings (the gears like icon), then
* CI/CD, then
* Runners, and click on the "Expand" button, then
* "Runners activated for this project", then
* Click on the "Edit" icon (next to the "Lock" Icon)
Tags are very important as they are used to route specific jobs to
specific types of runners, so it's a good idea to double check that
the automatically created tags are consistent with the OS and
architecture. For instance, an Ubuntu 20.04 aarch64 system should
have tags set as::
ubuntu_20.04,aarch64
Because the job definition at ``.gitlab-ci.d/custom-runners.yml``
would contain::
ubuntu-20.04-aarch64-all:
tags:
- ubuntu_20.04
- aarch64
It's also recommended to:
* increase the "Maximum job timeout" to something like ``2h``
* give it a better Description

14
docs/devel/testing/ci.rst Normal file
View file

@ -0,0 +1,14 @@
.. _ci:
==
CI
==
Most of QEMU's CI is run on GitLab's infrastructure although a number
of other CI services are used for specialised purposes. The most up to
date information about them and their status can be found on the
`project wiki testing page <https://wiki.qemu.org/Testing/CI>`_.
.. include:: ci-definitions.rst.inc
.. include:: ci-jobs.rst.inc
.. include:: ci-runners.rst.inc

304
docs/devel/testing/fuzzing.rst Normal file
View file

@ -0,0 +1,304 @@
========
Fuzzing
========
This document describes the virtual-device fuzzing infrastructure in QEMU and
how to use it to implement additional fuzzers.
Basics
------
Fuzzing operates by passing inputs to an entry point/target function. The
fuzzer tracks the code coverage triggered by the input. Based on these
findings, the fuzzer mutates the input and repeats the fuzzing.
To fuzz QEMU, we rely on libfuzzer. Unlike other fuzzers such as AFL, libfuzzer
is an *in-process* fuzzer. For the developer, this means that it is their
responsibility to ensure that state is reset between fuzzing-runs.
Building the fuzzers
--------------------
To build the fuzzers, install a recent version of clang:
Configure with (substitute the clang binaries with the version you installed).
Here, enable-sanitizers, is optional but it allows us to reliably detect bugs
such as out-of-bounds accesses, use-after-frees, double-frees etc.::
CC=clang-8 CXX=clang++-8 /path/to/configure --enable-fuzzing \
--enable-sanitizers
Fuzz targets are built similarly to system targets::
make qemu-fuzz-i386
This builds ``./qemu-fuzz-i386``
The first option to this command is: ``--fuzz-target=FUZZ_NAME``
To list all of the available fuzzers run ``qemu-fuzz-i386`` with no arguments.
For example::
./qemu-fuzz-i386 --fuzz-target=virtio-scsi-fuzz
Internally, libfuzzer parses all arguments that do not begin with ``"--"``.
Information about these is available by passing ``-help=1``
Now the only thing left to do is wait for the fuzzer to trigger potential
crashes.
Useful libFuzzer flags
----------------------
As mentioned above, libFuzzer accepts some arguments. Passing ``-help=1`` will
list the available arguments. In particular, these arguments might be helpful:
* ``CORPUS_DIR/`` : Specify a directory as the last argument to libFuzzer.
libFuzzer stores each "interesting" input in this corpus directory. The next
time you run libFuzzer, it will read all of the inputs from the corpus, and
continue fuzzing from there. You can also specify multiple directories.
libFuzzer loads existing inputs from all specified directories, but will only
write new ones to the first one specified.
* ``-max_len=4096`` : specify the maximum byte-length of the inputs libFuzzer
will generate.
* ``-close_fd_mask={1,2,3}`` : close, stderr, or both. Useful for targets that
trigger many debug/error messages, or create output on the serial console.
* ``-jobs=4 -workers=4`` : These arguments configure libFuzzer to run 4 fuzzers in
parallel (4 fuzzing jobs in 4 worker processes). Alternatively, with only
``-jobs=N``, libFuzzer automatically spawns a number of workers less than or equal
to half the available CPU cores. Replace 4 with a number appropriate for your
machine. Make sure to specify a ``CORPUS_DIR``, which will allow the parallel
fuzzers to share information about the interesting inputs they find.
* ``-use_value_profile=1`` : For each comparison operation, libFuzzer computes
``(caller_pc&4095) | (popcnt(Arg1 ^ Arg2) << 12)`` and places this in the
coverage table. Useful for targets with "magic" constants. If Arg1 came from
the fuzzer's input and Arg2 is a magic constant, then each time the Hamming
distance between Arg1 and Arg2 decreases, libFuzzer adds the input to the
corpus.
* ``-shrink=1`` : Tries to make elements of the corpus "smaller". Might lead to
better coverage performance, depending on the target.
Note that libFuzzer's exact behavior will depend on the version of
clang and libFuzzer used to build the device fuzzers.
Generating Coverage Reports
---------------------------
Code coverage is a crucial metric for evaluating a fuzzer's performance.
libFuzzer's output provides a "cov: " column that provides a total number of
unique blocks/edges covered. To examine coverage on a line-by-line basis we
can use Clang coverage:
1. Configure libFuzzer to store a corpus of all interesting inputs (see
CORPUS_DIR above)
2. ``./configure`` the QEMU build with ::
--enable-fuzzing \
--extra-cflags="-fprofile-instr-generate -fcoverage-mapping"
3. Re-run the fuzzer. Specify $CORPUS_DIR/* as an argument, telling libfuzzer
to execute all of the inputs in $CORPUS_DIR and exit. Once the process
exits, you should find a file, "default.profraw" in the working directory.
4. Execute these commands to generate a detailed HTML coverage-report::
llvm-profdata merge -output=default.profdata default.profraw
llvm-cov show ./path/to/qemu-fuzz-i386 -instr-profile=default.profdata \
--format html -output-dir=/path/to/output/report
Adding a new fuzzer
-------------------
Coverage over virtual devices can be improved by adding additional fuzzers.
Fuzzers are kept in ``tests/qtest/fuzz/`` and should be added to
``tests/qtest/fuzz/meson.build``
Fuzzers can rely on both qtest and libqos to communicate with virtual devices.
1. Create a new source file. For example ``tests/qtest/fuzz/foo-device-fuzz.c``.
2. Write the fuzzing code using the libqtest/libqos API. See existing fuzzers
for reference.
3. Add the fuzzer to ``tests/qtest/fuzz/meson.build``.
Fuzzers can be more-or-less thought of as special qtest programs which can
modify the qtest commands and/or qtest command arguments based on inputs
provided by libfuzzer. Libfuzzer passes a byte array and length. Commonly the
fuzzer loops over the byte-array interpreting it as a list of qtest commands,
addresses, or values.
The Generic Fuzzer
------------------
Writing a fuzz target can be a lot of effort (especially if a device driver has
not be built-out within libqos). Many devices can be fuzzed to some degree,
without any device-specific code, using the generic-fuzz target.
The generic-fuzz target is capable of fuzzing devices over their PIO, MMIO,
and DMA input-spaces. To apply the generic-fuzz to a device, we need to define
two env-variables, at minimum:
* ``QEMU_FUZZ_ARGS=`` is the set of QEMU arguments used to configure a machine, with
the device attached. For example, if we want to fuzz the virtio-net device
attached to a pc-i440fx machine, we can specify::
QEMU_FUZZ_ARGS="-M pc -nodefaults -netdev user,id=user0 \
-device virtio-net,netdev=user0"
* ``QEMU_FUZZ_OBJECTS=`` is a set of space-delimited strings used to identify
the MemoryRegions that will be fuzzed. These strings are compared against
MemoryRegion names and MemoryRegion owner names, to decide whether each
MemoryRegion should be fuzzed. These strings support globbing. For the
virtio-net example, we could use one of ::
QEMU_FUZZ_OBJECTS='virtio-net'
QEMU_FUZZ_OBJECTS='virtio*'
QEMU_FUZZ_OBJECTS='virtio* pcspk' # Fuzz the virtio devices and the speaker
QEMU_FUZZ_OBJECTS='*' # Fuzz the whole machine``
The ``"info mtree"`` and ``"info qom-tree"`` monitor commands can be especially
useful for identifying the ``MemoryRegion`` and ``Object`` names used for
matching.
As a generic rule-of-thumb, the more ``MemoryRegions``/Devices we match, the
greater the input-space, and the smaller the probability of finding crashing
inputs for individual devices. As such, it is usually a good idea to limit the
fuzzer to only a few ``MemoryRegions``.
To ensure that these env variables have been configured correctly, we can use::
./qemu-fuzz-i386 --fuzz-target=generic-fuzz -runs=0
The output should contain a complete list of matched MemoryRegions.
OSS-Fuzz
--------
QEMU is continuously fuzzed on `OSS-Fuzz
<https://github.com/google/oss-fuzz>`_. By default, the OSS-Fuzz build
will try to fuzz every fuzz-target. Since the generic-fuzz target
requires additional information provided in environment variables, we
pre-define some generic-fuzz configs in
``tests/qtest/fuzz/generic_fuzz_configs.h``. Each config must specify:
- ``.name``: To identify the fuzzer config
- ``.args`` OR ``.argfunc``: A string or pointer to a function returning a
string. These strings are used to specify the ``QEMU_FUZZ_ARGS``
environment variable. ``argfunc`` is useful when the config relies on e.g.
a dynamically created temp directory, or a free tcp/udp port.
- ``.objects``: A string that specifies the ``QEMU_FUZZ_OBJECTS`` environment
variable.
To fuzz additional devices/device configuration on OSS-Fuzz, send patches for
either a new device-specific fuzzer or a new generic-fuzz config.
Build details:
- The Dockerfile that sets up the environment for building QEMU's
fuzzers on OSS-Fuzz can be fund in the OSS-Fuzz repository
__(https://github.com/google/oss-fuzz/blob/master/projects/qemu/Dockerfile)
- The script responsible for building the fuzzers can be found in the
QEMU source tree at ``scripts/oss-fuzz/build.sh``
Building Crash Reproducers
-----------------------------------------
When we find a crash, we should try to create an independent reproducer, that
can be used on a non-fuzzer build of QEMU. This filters out any potential
false-positives, and improves the debugging experience for developers.
Here are the steps for building a reproducer for a crash found by the
generic-fuzz target.
- Ensure the crash reproduces::
qemu-fuzz-i386 --fuzz-target... ./crash-...
- Gather the QTest output for the crash::
QEMU_FUZZ_TIMEOUT=0 QTEST_LOG=1 FUZZ_SERIALIZE_QTEST=1 \
qemu-fuzz-i386 --fuzz-target... ./crash-... &> /tmp/trace
- Reorder and clean-up the resulting trace::
scripts/oss-fuzz/reorder_fuzzer_qtest_trace.py /tmp/trace > /tmp/reproducer
- Get the arguments needed to start qemu, and provide a path to qemu::
less /tmp/trace # The args should be logged at the top of this file
export QEMU_ARGS="-machine ..."
export QEMU_PATH="path/to/qemu-system"
- Ensure the crash reproduces in qemu-system::
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer
- From the crash output, obtain some string that identifies the crash. This
can be a line in the stack-trace, for example::
export CRASH_TOKEN="hw/usb/hcd-xhci.c:1865"
- Minimize the reproducer::
scripts/oss-fuzz/minimize_qtest_trace.py -M1 -M2 \
/tmp/reproducer /tmp/reproducer-minimized
- Confirm that the minimized reproducer still crashes::
$QEMU_PATH $QEMU_ARGS -qtest stdio < /tmp/reproducer-minimized
- Create a one-liner reproducer that can be sent over email::
./scripts/oss-fuzz/output_reproducer.py -bash /tmp/reproducer-minimized
- Output the C source code for a test case that will reproduce the bug::
./scripts/oss-fuzz/output_reproducer.py -owner "John Smith <john@smith.com>"\
-name "test_function_name" /tmp/reproducer-minimized
- Report the bug and send a patch with the C reproducer upstream
Implementation Details / Fuzzer Lifecycle
-----------------------------------------
The fuzzer has two entrypoints that libfuzzer calls. libfuzzer provides it's
own ``main()``, which performs some setup, and calls the entrypoints:
``LLVMFuzzerInitialize``: called prior to fuzzing. Used to initialize all of the
necessary state
``LLVMFuzzerTestOneInput``: called for each fuzzing run. Processes the input and
resets the state at the end of each run.
In more detail:
``LLVMFuzzerInitialize`` parses the arguments to the fuzzer (must start with two
dashes, so they are ignored by libfuzzer ``main()``). Currently, the arguments
select the fuzz target. Then, the qtest client is initialized. If the target
requires qos, qgraph is set up and the QOM/LIBQOS modules are initialized.
Then the QGraph is walked and the QEMU cmd_line is determined and saved.
After this, the ``vl.c:main`` is called to set up the guest. There are
target-specific hooks that can be called before and after main, for
additional setup(e.g. PCI setup, or VM snapshotting).
``LLVMFuzzerTestOneInput``: Uses qtest/qos functions to act based on the fuzz
input. It is also responsible for manually calling ``main_loop_wait`` to ensure
that bottom halves are executed and any cleanup required before the next input.
Since the same process is reused for many fuzzing runs, QEMU state needs to
be reset at the end of each run. For example, this can be done by rebooting the
VM, after each run.
- *Pros*: Straightforward and fast for simple fuzz targets.
- *Cons*: Depending on the device, does not reset all device state. If the
device requires some initialization prior to being ready for fuzzing (common
for QOS-based targets), this initialization needs to be done after each
reboot.
- *Example target*: ``i440fx-qtest-reboot-fuzz``

14
docs/devel/testing/index.rst Normal file
View file

@ -0,0 +1,14 @@
Testing QEMU
------------
Details about how to test QEMU and how it is integrated into our CI
testing infrastructure.
.. toctree::
:maxdepth: 3
main
qtest
acpi-bits
ci
fuzzing

1557
docs/devel/testing/main.rst Normal file
View file

File diff suppressed because it is too large Load diff

628
docs/devel/testing/qgraph.rst Normal file
View file

@ -0,0 +1,628 @@
.. _qgraph:
Qtest Driver Framework
======================
In order to test a specific driver, plain libqos tests need to
take care of booting QEMU with the right machine and devices.
This makes each test "hardcoded" for a specific configuration, reducing
the possible coverage that it can reach.
For example, the sdhci device is supported on both x86_64 and ARM boards,
therefore a generic sdhci test should test all machines and drivers that
support that device.
Using only libqos APIs, the test has to manually take care of
covering all the setups, and build the correct command line.
This also introduces backward compatibility issues: if a device/driver command
line name is changed, all tests that use that will not work
properly anymore and need to be adjusted.
The aim of qgraph is to create a graph of drivers, machines and tests such that
a test aimed to a certain driver does not have to care of
booting the right QEMU machine, pick the right device, build the command line
and so on. Instead, it only defines what type of device it is testing
(interface in qgraph terms) and the framework takes care of
covering all supported types of devices and machine architectures.
Following the above example, an interface would be ``sdhci``,
so the sdhci-test should only care of linking its qgraph node with
that interface. In this way, if the command line of a sdhci driver
is changed, only the respective qgraph driver node has to be adjusted.
QGraph concepts
---------------
The graph is composed by nodes that represent machines, drivers, tests
and edges that define the relationships between them (``CONSUMES``, ``PRODUCES``, and
``CONTAINS``).
Nodes
~~~~~
A node can be of four types:
- **QNODE_MACHINE**: for example ``arm/raspi2b``
- **QNODE_DRIVER**: for example ``generic-sdhci``
- **QNODE_INTERFACE**: for example ``sdhci`` (interface for all ``-sdhci``
drivers).
An interface is not explicitly created, it will be automatically
instantiated when a node consumes or produces it.
An interface is simply a struct that abstracts the various drivers
for the same type of device, and offers an API to the nodes that
use it ("consume" relation in qgraph terms) that is implemented/backed up by the drivers that implement it ("produce" relation in qgraph terms).
- **QNODE_TEST**: for example ``sdhci-test``. A test consumes an interface
and tests the functions provided by it.
Notes for the nodes:
- QNODE_MACHINE: each machine struct must have a ``QGuestAllocator`` and
implement ``get_driver()`` to return the allocator mapped to the interface
"memory". The function can also return ``NULL`` if the allocator
is not set.
- QNODE_DRIVER: driver names must be unique, and machines and nodes
planned to be "consumed" by other nodes must match QEMU
drivers name, otherwise they won't be discovered
Edges
~~~~~
An edge relation between two nodes (drivers or machines) ``X`` and ``Y`` can be:
- ``X CONSUMES Y``: ``Y`` can be plugged into ``X``
- ``X PRODUCES Y``: ``X`` provides the interface ``Y``
- ``X CONTAINS Y``: ``Y`` is part of ``X`` component
Execution steps
~~~~~~~~~~~~~~~
The basic framework steps are the following:
- All nodes and edges are created in their respective
machine/driver/test files
- The framework starts QEMU and asks for a list of available devices
and machines (note that only machines and "consumed" nodes are mapped
1:1 with QEMU devices)
- The framework walks the graph starting from the available machines and
performs a Depth First Search for tests
- Once a test is found, the path is walked again and all drivers are
allocated accordingly and the final interface is passed to the test
- The test is executed
- Unused objects are cleaned and the path discovery is continued
Depending on the QEMU binary used, only some drivers/machines will be
available and only test that are reached by them will be executed.
Command line
~~~~~~~~~~~~
Command line is built by using node names and optional arguments
passed by the user when building the edges.
There are three types of command line arguments:
- ``in node`` : created from the node name. For example, machines will
have ``-M <machine>`` to its command line, while devices
``-device <device>``. It is automatically done by the framework.
- ``after node`` : added as additional argument to the node name.
This argument is added optionally when creating edges,
by setting the parameter ``after_cmd_line`` and
``extra_edge_opts`` in ``QOSGraphEdgeOptions``.
The framework automatically adds
a comma before ``extra_edge_opts``,
because it is going to add attributes
after the destination node pointed by
the edge containing these options, and automatically
adds a space before ``after_cmd_line``, because it
adds an additional device, not an attribute.
- ``before node`` : added as additional argument to the node name.
This argument is added optionally when creating edges,
by setting the parameter ``before_cmd_line`` in
``QOSGraphEdgeOptions``. This attribute
is going to add attributes before the destination node
pointed by the edge containing these options. It is
helpful to commands that are not node-representable,
such as ``-fdsev`` or ``-netdev``.
While adding command line in edges is always used, not all nodes names are
used in every path walk: this is because the contained or produced ones
are already added by QEMU, so only nodes that "consumes" will be used to
build the command line. Also, nodes that will have ``{ "abstract" : true }``
as QMP attribute will loose their command line, since they are not proper
devices to be added in QEMU.
Example::
QOSGraphEdgeOptions opts = {
.before_cmd_line = "-drive id=drv0,if=none,file=null-co://,"
"file.read-zeroes=on,format=raw",
.after_cmd_line = "-device scsi-hd,bus=vs0.0,drive=drv0",
opts.extra_device_opts = "id=vs0";
};
qos_node_create_driver("virtio-scsi-device",
virtio_scsi_device_create);
qos_node_consumes("virtio-scsi-device", "virtio-bus", &opts);
Will produce the following command line:
``-drive id=drv0,if=none,file=null-co://, -device virtio-scsi-device,id=vs0 -device scsi-hd,bus=vs0.0,drive=drv0``
Troubleshooting unavailable tests
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
If there is no path from an available machine to a test then that test will be
unavailable and won't execute. This can happen if a test or driver did not set
up its qgraph node correctly. It can also happen if the necessary machine type
or device is missing from the QEMU binary because it was compiled out or
otherwise.
It is possible to troubleshoot unavailable tests by running::
$ QTEST_QEMU_BINARY=build/qemu-system-x86_64 build/tests/qtest/qos-test --verbose
# ALL QGRAPH EDGES: {
# src='virtio-net'
# |-> dest='virtio-net-tests/vhost-user/multiqueue' type=2 (node=0x559142109e30)
# |-> dest='virtio-net-tests/vhost-user/migrate' type=2 (node=0x559142109d00)
# src='virtio-net-pci'
# |-> dest='virtio-net' type=1 (node=0x55914210d740)
# src='pci-bus'
# |-> dest='virtio-net-pci' type=2 (node=0x55914210d880)
# src='pci-bus-pc'
# |-> dest='pci-bus' type=1 (node=0x559142103f40)
# src='i440FX-pcihost'
# |-> dest='pci-bus-pc' type=0 (node=0x55914210ac70)
# src='x86_64/pc'
# |-> dest='i440FX-pcihost' type=0 (node=0x5591421117f0)
# src=''
# |-> dest='x86_64/pc' type=0 (node=0x559142111600)
# |-> dest='arm/raspi2b' type=0 (node=0x559142110740)
...
# }
# ALL QGRAPH NODES: {
# name='virtio-net-tests/announce-self' type=3 cmd_line='(null)' [available]
# name='arm/raspi2b' type=0 cmd_line='-M raspi2b ' [UNAVAILABLE]
...
# }
The ``virtio-net-tests/announce-self`` test is listed as "available" in the
"ALL QGRAPH NODES" output. This means the test will execute. We can follow the
qgraph path in the "ALL QGRAPH EDGES" output as follows: '' -> 'x86_64/pc' ->
'i440FX-pcihost' -> 'pci-bus-pc' -> 'pci-bus' -> 'virtio-net-pci' ->
'virtio-net'. The root of the qgraph is '' and the depth first search begins
there.
The ``arm/raspi2b`` machine node is listed as "UNAVAILABLE". Although it is
reachable from the root via '' -> 'arm/raspi2b' the node is unavailable because
the QEMU binary did not list it when queried by the framework. This is expected
because we used the ``qemu-system-x86_64`` binary which does not support ARM
machine types.
If a test is unexpectedly listed as "UNAVAILABLE", first check that the "ALL
QGRAPH EDGES" output reports edge connectivity from the root ('') to the test.
If there is no connectivity then the qgraph nodes were not set up correctly and
the driver or test code is incorrect. If there is connectivity, check the
availability of each node in the path in the "ALL QGRAPH NODES" output. The
first unavailable node in the path is the reason why the test is unavailable.
Typically this is because the QEMU binary lacks support for the necessary
machine type or device.
Creating a new driver and its interface
---------------------------------------
Here we continue the ``sdhci`` use case, with the following scenario:
- ``sdhci-test`` aims to test the ``read[q,w], writeq`` functions
offered by the ``sdhci`` drivers.
- The current ``sdhci`` device is supported by both ``x86_64/pc`` and ``ARM``
(in this example we focus on the ``arm-raspi2b``) machines.
- QEMU offers 2 types of drivers: ``QSDHCI_MemoryMapped`` for ``ARM`` and
``QSDHCI_PCI`` for ``x86_64/pc``. Both implement the
``read[q,w], writeq`` functions.
In order to implement such scenario in qgraph, the test developer needs to:
- Create the ``x86_64/pc`` machine node. This machine uses the
``pci-bus`` architecture so it ``contains`` a PCI driver,
``pci-bus-pc``. The actual path is
``x86_64/pc --contains--> 1440FX-pcihost --contains-->
pci-bus-pc --produces--> pci-bus``.
For the sake of this example,
we do not focus on the PCI interface implementation.
- Create the ``sdhci-pci`` driver node, representing ``QSDHCI_PCI``.
The driver uses the PCI bus (and its API),
so it must ``consume`` the ``pci-bus`` generic interface (which abstracts
all the pci drivers available)
``sdhci-pci --consumes--> pci-bus``
- Create an ``arm/raspi2b`` machine node. This machine ``contains``
a ``generic-sdhci`` memory mapped ``sdhci`` driver node, representing
``QSDHCI_MemoryMapped``.
``arm/raspi2b --contains--> generic-sdhci``
- Create the ``sdhci`` interface node. This interface offers the
functions that are shared by all ``sdhci`` devices.
The interface is produced by ``sdhci-pci`` and ``generic-sdhci``,
the available architecture-specific drivers.
``sdhci-pci --produces--> sdhci``
``generic-sdhci --produces--> sdhci``
- Create the ``sdhci-test`` test node. The test ``consumes`` the
``sdhci`` interface, using its API. It doesn't need to look at
the supported machines or drivers.
``sdhci-test --consumes--> sdhci``
``arm-raspi2b`` machine, simplified from
``tests/qtest/libqos/arm-raspi2-machine.c``::
#include "qgraph.h"
struct QRaspi2Machine {
QOSGraphObject obj;
QGuestAllocator alloc;
QSDHCI_MemoryMapped sdhci;
};
static void *raspi2_get_driver(void *object, const char *interface)
{
QRaspi2Machine *machine = object;
if (!g_strcmp0(interface, "memory")) {
return &machine->alloc;
}
fprintf(stderr, "%s not present in arm/raspi2b\n", interface);
g_assert_not_reached();
}
static QOSGraphObject *raspi2_get_device(void *obj,
const char *device)
{
QRaspi2Machine *machine = obj;
if (!g_strcmp0(device, "generic-sdhci")) {
return &machine->sdhci.obj;
}
fprintf(stderr, "%s not present in arm/raspi2b\n", device);
g_assert_not_reached();
}
static void *qos_create_machine_arm_raspi2(QTestState *qts)
{
QRaspi2Machine *machine = g_new0(QRaspi2Machine, 1);
alloc_init(&machine->alloc, ...);
/* Get node(s) contained inside (CONTAINS) */
machine->obj.get_device = raspi2_get_device;
/* Get node(s) produced (PRODUCES) */
machine->obj.get_driver = raspi2_get_driver;
/* free the object */
machine->obj.destructor = raspi2_destructor;
qos_init_sdhci_mm(&machine->sdhci, ...);
return &machine->obj;
}
static void raspi2_register_nodes(void)
{
/* arm/raspi2b --contains--> generic-sdhci */
qos_node_create_machine("arm/raspi2b",
qos_create_machine_arm_raspi2);
qos_node_contains("arm/raspi2b", "generic-sdhci", NULL);
}
libqos_init(raspi2_register_nodes);
``x86_64/pc`` machine, simplified from
``tests/qtest/libqos/x86_64_pc-machine.c``::
#include "qgraph.h"
struct i440FX_pcihost {
QOSGraphObject obj;
QPCIBusPC pci;
};
struct QX86PCMachine {
QOSGraphObject obj;
QGuestAllocator alloc;
i440FX_pcihost bridge;
};
/* i440FX_pcihost */
static QOSGraphObject *i440FX_host_get_device(void *obj,
const char *device)
{
i440FX_pcihost *host = obj;
if (!g_strcmp0(device, "pci-bus-pc")) {
return &host->pci.obj;
}
fprintf(stderr, "%s not present in i440FX-pcihost\n", device);
g_assert_not_reached();
}
/* x86_64/pc machine */
static void *pc_get_driver(void *object, const char *interface)
{
QX86PCMachine *machine = object;
if (!g_strcmp0(interface, "memory")) {
return &machine->alloc;
}
fprintf(stderr, "%s not present in x86_64/pc\n", interface);
g_assert_not_reached();
}
static QOSGraphObject *pc_get_device(void *obj, const char *device)
{
QX86PCMachine *machine = obj;
if (!g_strcmp0(device, "i440FX-pcihost")) {
return &machine->bridge.obj;
}
fprintf(stderr, "%s not present in x86_64/pc\n", device);
g_assert_not_reached();
}
static void *qos_create_machine_pc(QTestState *qts)
{
QX86PCMachine *machine = g_new0(QX86PCMachine, 1);
/* Get node(s) contained inside (CONTAINS) */
machine->obj.get_device = pc_get_device;
/* Get node(s) produced (PRODUCES) */
machine->obj.get_driver = pc_get_driver;
/* free the object */
machine->obj.destructor = pc_destructor;
pc_alloc_init(&machine->alloc, qts, ALLOC_NO_FLAGS);
/* Get node(s) contained inside (CONTAINS) */
machine->bridge.obj.get_device = i440FX_host_get_device;
return &machine->obj;
}
static void pc_machine_register_nodes(void)
{
/* x86_64/pc --contains--> 1440FX-pcihost --contains-->
* pci-bus-pc [--produces--> pci-bus (in pci.h)] */
qos_node_create_machine("x86_64/pc", qos_create_machine_pc);
qos_node_contains("x86_64/pc", "i440FX-pcihost", NULL);
/* contained drivers don't need a constructor,
* they will be init by the parent */
qos_node_create_driver("i440FX-pcihost", NULL);
qos_node_contains("i440FX-pcihost", "pci-bus-pc", NULL);
}
libqos_init(pc_machine_register_nodes);
``sdhci`` taken from ``tests/qtest/libqos/sdhci.c``::
/* Interface node, offers the sdhci API */
struct QSDHCI {
uint16_t (*readw)(QSDHCI *s, uint32_t reg);
uint64_t (*readq)(QSDHCI *s, uint32_t reg);
void (*writeq)(QSDHCI *s, uint32_t reg, uint64_t val);
/* other fields */
};
/* Memory Mapped implementation of QSDHCI */
struct QSDHCI_MemoryMapped {
QOSGraphObject obj;
QSDHCI sdhci;
/* other driver-specific fields */
};
/* PCI implementation of QSDHCI */
struct QSDHCI_PCI {
QOSGraphObject obj;
QSDHCI sdhci;
/* other driver-specific fields */
};
/* Memory mapped implementation of QSDHCI */
static void *sdhci_mm_get_driver(void *obj, const char *interface)
{
QSDHCI_MemoryMapped *smm = obj;
if (!g_strcmp0(interface, "sdhci")) {
return &smm->sdhci;
}
fprintf(stderr, "%s not present in generic-sdhci\n", interface);
g_assert_not_reached();
}
void qos_init_sdhci_mm(QSDHCI_MemoryMapped *sdhci, QTestState *qts,
uint32_t addr, QSDHCIProperties *common)
{
/* Get node contained inside (CONTAINS) */
sdhci->obj.get_driver = sdhci_mm_get_driver;
/* SDHCI interface API */
sdhci->sdhci.readw = sdhci_mm_readw;
sdhci->sdhci.readq = sdhci_mm_readq;
sdhci->sdhci.writeq = sdhci_mm_writeq;
sdhci->qts = qts;
}
/* PCI implementation of QSDHCI */
static void *sdhci_pci_get_driver(void *object,
const char *interface)
{
QSDHCI_PCI *spci = object;
if (!g_strcmp0(interface, "sdhci")) {
return &spci->sdhci;
}
fprintf(stderr, "%s not present in sdhci-pci\n", interface);
g_assert_not_reached();
}
static void *sdhci_pci_create(void *pci_bus,
QGuestAllocator *alloc,
void *addr)
{
QSDHCI_PCI *spci = g_new0(QSDHCI_PCI, 1);
QPCIBus *bus = pci_bus;
uint64_t barsize;
qpci_device_init(&spci->dev, bus, addr);
/* SDHCI interface API */
spci->sdhci.readw = sdhci_pci_readw;
spci->sdhci.readq = sdhci_pci_readq;
spci->sdhci.writeq = sdhci_pci_writeq;
/* Get node(s) produced (PRODUCES) */
spci->obj.get_driver = sdhci_pci_get_driver;
spci->obj.start_hw = sdhci_pci_start_hw;
spci->obj.destructor = sdhci_destructor;
return &spci->obj;
}
static void qsdhci_register_nodes(void)
{
QOSGraphEdgeOptions opts = {
.extra_device_opts = "addr=04.0",
};
/* generic-sdhci */
/* generic-sdhci --produces--> sdhci */
qos_node_create_driver("generic-sdhci", NULL);
qos_node_produces("generic-sdhci", "sdhci");
/* sdhci-pci */
/* sdhci-pci --produces--> sdhci
* sdhci-pci --consumes--> pci-bus */
qos_node_create_driver("sdhci-pci", sdhci_pci_create);
qos_node_produces("sdhci-pci", "sdhci");
qos_node_consumes("sdhci-pci", "pci-bus", &opts);
}
libqos_init(qsdhci_register_nodes);
In the above example, all possible types of relations are created::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci --consumes--> pci-bus <--produces--+
|
+--produces--+
|
v
sdhci
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
or inverting the consumes edge in consumed_by::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
+--produces--+
|
v
sdhci
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
Adding a new test
-----------------
Given the above setup, adding a new test is very simple.
``sdhci-test``, taken from ``tests/qtest/sdhci-test.c``::
static void check_capab_sdma(QSDHCI *s, bool supported)
{
uint64_t capab, capab_sdma;
capab = s->readq(s, SDHC_CAPAB);
capab_sdma = FIELD_EX64(capab, SDHC_CAPAB, SDMA);
g_assert_cmpuint(capab_sdma, ==, supported);
}
static void test_registers(void *obj, void *data,
QGuestAllocator *alloc)
{
QSDHCI *s = obj;
/* example test */
check_capab_sdma(s, s->props.capab.sdma);
}
static void register_sdhci_test(void)
{
/* sdhci-test --consumes--> sdhci */
qos_add_test("registers", "sdhci", test_registers, NULL);
}
libqos_init(register_sdhci_test);
Here a new test is created, consuming ``sdhci`` interface node
and creating a valid path from both machines to a test.
Final graph will be like this::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci --consumes--> pci-bus <--produces--+
|
+--produces--+
|
v
sdhci <--consumes-- sdhci-test
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
or inverting the consumes edge in consumed_by::
x86_64/pc --contains--> 1440FX-pcihost --contains--> pci-bus-pc
|
sdhci-pci <--consumed by-- pci-bus <--produces--+
|
+--produces--+
|
v
sdhci --consumed by--> sdhci-test
^
|
+--produces-- +
|
arm/raspi2b --contains--> generic-sdhci
Assuming there the binary is
``QTEST_QEMU_BINARY=./qemu-system-x86_64``
a valid test path will be:
``/x86_64/pc/1440FX-pcihost/pci-bus-pc/pci-bus/sdhci-pc/sdhci/sdhci-test``
and for the binary ``QTEST_QEMU_BINARY=./qemu-system-arm``:
``/arm/raspi2b/generic-sdhci/sdhci/sdhci-test``
Additional examples are also in ``test-qgraph.c``
Qgraph API reference
--------------------
.. kernel-doc:: tests/qtest/libqos/qgraph.h

91
docs/devel/testing/qtest.rst Normal file
View file

@ -0,0 +1,91 @@
========================================
QTest Device Emulation Testing Framework
========================================
.. toctree::
qgraph
QTest is a device emulation testing framework. It can be very useful to test
device models; it could also control certain aspects of QEMU (such as virtual
clock stepping), with a special purpose "qtest" protocol. Refer to
:ref:`qtest-protocol` for more details of the protocol.
QTest cases can be executed with
.. code::
make check-qtest
The QTest library is implemented by ``tests/qtest/libqtest.c`` and the API is
defined in ``tests/qtest/libqtest.h``.
Consider adding a new QTest case when you are introducing a new virtual
hardware, or extending one if you are adding functionalities to an existing
virtual device.
On top of libqtest, a higher level library, ``libqos``, was created to
encapsulate common tasks of device drivers, such as memory management and
communicating with system buses or devices. Many virtual device tests use
libqos instead of directly calling into libqtest.
Libqos also offers the Qgraph API to increase each test coverage and
automate QEMU command line arguments and devices setup.
Refer to :ref:`qgraph` for Qgraph explanation and API.
Steps to add a new QTest case are:
1. Create a new source file for the test. (More than one file can be added as
necessary.) For example, ``tests/qtest/foo-test.c``.
2. Write the test code with the glib and libqtest/libqos API. See also existing
tests and the library headers for reference.
3. Register the new test in ``tests/qtest/meson.build``. Add the test
executable name to an appropriate ``qtests_*`` variable. There is
one variable per architecture, plus ``qtests_generic`` for tests
that can be run for all architectures. For example::
qtests_generic = [
...
'foo-test',
...
]
4. If the test has more than one source file or needs to be linked with any
dependency other than ``qemuutil`` and ``qos``, list them in the ``qtests``
dictionary. For example a test that needs to use the ``QIO`` library
will have an entry like::
{
...
'foo-test': [io],
...
}
Debugging a QTest failure is slightly harder than the unit test because the
tests look up QEMU program names in the environment variables, such as
``QTEST_QEMU_BINARY`` and ``QTEST_QEMU_IMG``, and also because it is not easy
to attach gdb to the QEMU process spawned from the test. But manual invoking
and using gdb on the test is still simple to do: find out the actual command
from the output of
.. code::
make check-qtest V=1
which you can run manually.
.. _qtest-protocol:
QTest Protocol
--------------
.. kernel-doc:: system/qtest.c
:doc: QTest Protocol
libqtest API reference
----------------------
.. kernel-doc:: tests/qtest/libqtest.h