motorhead/qemu
1
0
Fork 0
mirror of https://github.com/Motorhead1991/qemu.git synced 2025-08-04 00:03:54 -06:00
Code Issues Projects Releases Packages Wiki Activity Actions

docs/devel: rationalise unstable gitlab tests under FLAKY_TESTS

Browse source 
It doesn't make sense to have two classes of flaky tests. While it may
take the constrained environment of CI to trigger failures easily it
doesn't mean they don't occasionally happen on developer machines. As
CI is the gating factor to passing there is no point developers
running the tests locally anyway unless they are trying to fix things.

While we are at it update the language in the docs to discourage the
QEMU_TEST_FLAKY_TESTS becoming a permanent solution.

Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20231201093633.2551497-3-alex.bennee@linaro.org>
This commit is contained in:
Alex Bennée 2023-12-01 09:36:28 +00:00
parent 5dcf6334ae
commit 9b45cc9931
13 changed files with 71 additions and 44 deletions
Download patch file Download diff file Expand all files Collapse all files

37
docs/devel/testing.rst
View file

@ -1371,23 +1371,32 @@ conditions. For example, tests that take longer to execute when QEMU is
compiled with debug flags. Therefore, the ``AVOCADO_TIMEOUT_EXPECTED`` variable
has been used to determine whether those tests should run or not.
GITLAB_CI
^^^^^^^^^
A number of tests are flagged to not run on the GitLab CI. Usually because
they proved to the flaky or there are constraints on the CI environment which
would make them fail. If you encounter a similar situation then use that
variable as shown on the code snippet below to skip the test:
.. code::
@skipIf(os.getenv('GITLAB_CI'), 'Running on GitLab')
def test(self):
do_something()
QEMU_TEST_FLAKY_TESTS
^^^^^^^^^^^^^^^^^^^^^
Some tests are not working reliably and thus are disabled by default.
Set this environment variable to enable them.
This includes tests that don't run reliably on GitLab's CI which
usually expose real issues that are rarely seen on developer machines
due to the constraints of the CI environment. If you encounter a
similar situation then raise a bug and then mark the test as shown on
the code snippet below:
.. code::
# See https://gitlab.com/qemu-project/qemu/-/issues/nnnn
@skipUnless(os.getenv('QEMU_TEST_FLAKY_TESTS'), 'Test is unstable on GitLab')
def test(self):
do_something()
Tests should not live in this state forever and should either be fixed
or eventually removed.
To run such tests locally you will need to set the environment
variable. For example:
.. code::
env QEMU_TEST_FLAKY_TESTS=1 ./pyvenv/bin/avocado run \
tests/avocado/boot_linux.py:BootLinuxPPC64.test_pseries_tcg
Uninstalling Avocado
~~~~~~~~~~~~~~~~~~~~