Now that all Avocado tests have been converted to or been replaced by
other functional tests, we can delete the remainders of the Avocado
tests from the QEMU source tree.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-16-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This file was meant for defining the vocabulary for our testing
efforts, but it did not age well: First, the definitions are not
only about the CI part, but also about testing in general, so most
of the information should rather reside in main.rst instead.
Second, some vocabulary has never been really adopted by the QEMU
project, for example we never really use the word "system testing"
since "system" rather means the system emulator binaries in the
QEMU project (and we also don't do any testing with other components
like libvirt and virt-managers here). It also defines that the qtests
are the "functional" tests in QEMU, which is not really up to date
anymore after the "tests/functional" framework has been introduced
a couple of months ago (FWIW, the qtests could rather be seen as a
mix between unit testing and functional testing).
To solve this problem, move the useful parts of this file into
main.rst and directly into ci.rst, and drop the ones (like "system
testing") that we don't really need anymore.
Message-ID: <20250314085959.1585568-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Since we don't run the Avocado jobs in the CI anymore, rename
these variables to QEMU_JOB_FUNCTIONAL and QEMU_CI_FUNCTIONAL.
Also, there was a mismatch between the documentation and the
implementation of QEMU_CI_AVOCADO_TESTING: While the documentation
said that you had to "Set this variable to have the tests using the
Avocado framework run automatically", you indeed needed to set it
to make the pipelines appear in your dashboard - but they were never
run automatically in forks and had to be triggered manually. Let's
improve this now: No need to hide these pipelines from the users
by default anymore (the functional tests should be stable enough
nowadays), and rather allow the users to run the pipelines auto-
matically with this switch now instead, as was documented.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250414113031.151105-15-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
We should tell the users where to find the log file.
While we're at it, also rename the "Overview" heading to a
more accurate "Introduction to writing tests" instead.
Reported-by: Aditya Gupta <adityag@linux.ibm.com>
Message-ID: <20250318092021.53719-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Seems 'Asset' got missed in the documentation by mistake.
Also fix the one spellcheck issue pointed by spellcheck
Signed-off-by: Aditya Gupta <adityag@linux.ibm.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20250306060706.1982992-1-adityag@linux.ibm.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The 'qemu_bin' field is currently set on the class, despite being
accessed as if it were an object instance field with 'self.qemu_bin'.
This is no obvious need to have it as a class field, so move it into
the object instance.
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20250228102738.3064045-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Some tests have a very long runtime and might run into timeout issues
e.g. when QEMU has been compiled with --enable-debug. Add a decorator
for marking them more easily. Rename the corresponding environment
variable to be more in sync with the other QEMU_TEST_ALLOW_* switches
that we already have, and add a paragraph about it in the documentation.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250128152839.184599-2-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Present the various parts of QEMU and organization of codebase.
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Message-Id: <20241209183104.365796-6-pierrick.bouvier@linaro.org>
[AJB: tweak commit summary, update MAINTAINERS]
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20250116160306.1709518-36-alex.bennee@linaro.org>
Make it clear that the commands have to be run from the folder with the
build, and use the python3 from our pyvenv to make sure that the
pycotap module is available.
Message-ID: <20241112115302.470527-1-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:
$ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
PYTHONPATH=./python \
./tests/functional/test_arm_tuxrun.py
And then, when the test stalls, in a second shell run:
$ ./scripts/qmp/qmp-shell backdoor.sock
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
The build/tests/functional subdirectories are consuming huge amounts
of disk space.
Split the location for scratch files into a 'scratch' sub-directory,
separate from log files, and delete it upon completion of each test.
The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve
this scratch dir for debugging access if required.
Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
Mention it's now possible to build with gcc, instead of clang, and
explain how to build a sanitized glib version.
Signed-off-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20240910174013.1433331-4-pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241023113406.1284676-6-alex.bennee@linaro.org>
Convert blkverify.txt to rST format.
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 20240816132212.3602106-3-peter.maydell@linaro.org
Convert blkdebug.txt to rST format. We put it into index-build.rst
because it falls under the "test" part of "QEMU Build and Test
System".
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-id: 20240816132212.3602106-2-peter.maydell@linaro.org
Replace the footnotes with inline links whenever the footnote text
consists of nothing but the URL. While at it, make the link texts
consistent in the surrounding areas, for example avoiding usage of
"here" for the link's text.
In the case of acpi-bits.rst this fixes a build failure with Sphinx
8.1.0, because the FOSDEM link was duplicated in the paragraph and the
new version is a lot stricter about unreferenced footnotes.
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
This options has been removed at cb771ac1f5 (meson: Split
--enable-sanitizers to --enable-{asan, ubsan}, 2024-08-13), so let's
update its last standing mention in the docs.
Signed-off-by: Matheus Tavares Bernardino <quic_mathbern@quicinc.com>
Reviewed-by: Brian Cain <bcain@quicinc.com>
Message-ID: <0ecf4e1ab26771009d74a2ce61e7c17ddc586ef7.1726226316.git.quic_mathbern@quicinc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
We do not always want both address and undefined behavior
sanitizers running at the same time.
For the gitlab custom-runners, drop to only --enable-ubsan.
These jobs are not run by default, but as will be obvious in the
next patch, we don't run ASan on x86 either, and it seems wrong
to hold aarch64 and s390x to a different standard.
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240813095216.306555-2-richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
I guess the same change came in via two patch series. Remove the
repetition.
Fixes: 2a851fca9f (docs/devel: remind developers to run CI container pipeline when updating images)
Reviewed-by: Pierrick Bouvier <pierrick.bouvier@linaro.org>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20240910173900.4154726-5-alex.bennee@linaro.org>
Document the new functional testing framework. The text is originally
based on the Avocado documentation, but heavily modified to match the
new framework.
Message-ID: <20240830133841.142644-45-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The avocado_qemu.Test class has been renamed a while back in commit
2283b627bc ("tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest"),
so we should reflect this now in the documentation, too.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-44-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
The main testing documentation file got very overloaded already.
Thus let's split the Avocado information into a separate file.
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240830133841.142644-43-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
Building and testing are two separate topics, so let's split the testing
into a separate category and move the related files into a separate folder.
Message-ID: <20240830133841.142644-42-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
