motorhead/qemu
1
0
Fork 0
mirror of https://github.com/Motorhead1991/qemu.git synced 2025-12-11 16:00:50 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

Integration testing patches

Browse source 
- Rename tests/acceptance/ -> tests/avocado/
 - Rename avocado_qemu.Test -> avocado_qemu.QemuSystemTest
 - Introduce QemuUserTest class
 - Add the first linux-user test, covering the bFLT loader
 -----BEGIN PGP SIGNATURE-----
 
 iQIzBAABCAAdFiEE+qvnXhKRciHc/Wuy4+MsLN6twN4FAmGJTW4ACgkQ4+MsLN6t
 wN7D0w/+NgsGxkruLDWNTK5lOFljh2L+oF/WjQLwK0e9wLDN+nRcTlBkHMMVFndC
 Rfaox26gcwv0HkpBGLWdmsVed7zeZwvh6nMQtE8upB2nf2ltyYG9Bi7dt352/B4L
 NjS8OT/hFvuszD4+aJFrkcCVKkPUfcWasoBBoy4mtJt/1TzH972uHYksRCjqPEwe
 bN88V8GJfW5xcVzCvsc63jsdmamWpPdrWgX1VcL8eBrkScoyhdokE56pZ0vGR4d5
 ecGekTbfnJWl9Wy60im9Upg29WcGfJRcWgxnRHgEU7E1WNaVmQ2UlndEmhveATnH
 UcztfItir8iS58FsaJpKxlAmxv+SrivrYWQ5bLz88tIWoU7m8o90kzmM69DYOFfK
 AP440p7N5nF1nz9NGUT+5P5rEIYNncnYuwKgUiqQ9p2FOYeO6F6i4894QyB6Ebwo
 WpUgroSmOnUFEdXOixwyZYVswhXHp7Zfws+g+JTjCmyyeD7qF8Dd1ZeNqREPiLk/
 2GDe757b07qzXriN6/7YXeIvkCFoe65mCxkFWha45C13wfLyqP+1wML5utj3YpdA
 vqU2soQzGdH56cebpmqxxmVSAuZOPKNgQpNJ7UR+r7gFHXMHuqrTDQ+wldFnEKjT
 s2VoAeHdXJmO/+WDQgKKJCrBYmCmlEwObYS+clkN8HZtsr0purk=
 =6oPo
 -----END PGP SIGNATURE-----

Merge remote-tracking branch 'remotes/philmd/tags/avocado-20211108' into staging

Integration testing patches

- Rename tests/acceptance/ -> tests/avocado/
- Rename avocado_qemu.Test -> avocado_qemu.QemuSystemTest
- Introduce QemuUserTest class
- Add the first linux-user test, covering the bFLT loader

# gpg: Signature made Mon 08 Nov 2021 05:16:46 PM CET
# gpg:                using RSA key FAABE75E12917221DCFD6BB2E3E32C2CDEADC0DE
# gpg: Good signature from "Philippe Mathieu-Daudé (F4BUG) <f4bug@amsat.org>" [full]

* remotes/philmd/tags/avocado-20211108:
  tests/avocado: Remove p7zip binary availability check
  tests/avocado: Rename avocado_qemu.Test -> QemuSystemTest
  tests/avocado: Add bFLT loader linux-user test
  tests/avocado: Share useful helpers from virtiofs_submounts test
  tests/avocado: Introduce QemuUserTest base class
  tests/avocado: Make pick_default_qemu_bin() more generic
  tests/avocado: Extract QemuBaseTest from Test
  tests/acceptance: rename tests acceptance to tests avocado
  tests/acceptance: introduce new check-avocado target

Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
This commit is contained in:
Richard Henderson 2021-11-08 18:50:09 +01:00
commit 114f3c8cc4
65 changed files with 364 additions and 284 deletions
Download patch file Download diff file Expand all files Collapse all files

2
docs/devel/build-system.rst
View file

@ -416,7 +416,7 @@ number of dynamically created files listed later.
``tests/Makefile.include``
Rules for external test harnesses. These include the TCG tests,
``qemu-iotests`` and the Avocado-based acceptance tests.
``qemu-iotests`` and the Avocado-based integration tests.
``tests/docker/Makefile.include``
Rules for Docker tests. Like tests/Makefile, this file is included

2
docs/devel/ci-definitions.rst.inc
View file

@ -59,7 +59,7 @@ to system testing [5]_. Note that, in some cases, system testing may require
interaction with third-party software, like operating system images, databases,
networks, and so on.
On QEMU, system testing is represented by the 'check-acceptance' target from
On QEMU, system testing is represented by the 'check-avocado' target from
'make'.
Flaky tests

57
docs/devel/testing.rst
View file

@ -653,17 +653,16 @@ supported. To start the fuzzer, run
Alternatively, some command different from "qemu-img info" can be tested, by
changing the ``-c`` option.
Acceptance tests using the Avocado Framework
--------------------------------------------
Integration tests using the Avocado Framework
---------------------------------------------
The ``tests/acceptance`` directory hosts functional tests, also known
as acceptance level tests. They're usually higher level tests, and
may interact with external resources and with various guest operating
systems.
The ``tests/avocado`` directory hosts integration tests. They're usually
higher level tests, and may interact with external resources and with
various guest operating systems.
These tests are written using the Avocado Testing Framework (which must
be installed separately) in conjunction with a the ``avocado_qemu.Test``
class, implemented at ``tests/acceptance/avocado_qemu``.
class, implemented at ``tests/avocado/avocado_qemu``.
Tests based on ``avocado_qemu.Test`` can easily:
@ -695,11 +694,11 @@ Tests based on ``avocado_qemu.Test`` can easily:
Running tests
~~~~~~~~~~~~~
You can run the acceptance tests simply by executing:
You can run the avocado tests simply by executing:
.. code::
make check-acceptance
make check-avocado
This involves the automatic creation of Python virtual environment
within the build tree (at ``tests/venv``) which will have all the
@ -714,12 +713,12 @@ specific version, they may be on packages named ``python3-venv`` and
``python3-pip``.
It is also possible to run tests based on tags using the
``make check-acceptance`` command and the ``AVOCADO_TAGS`` environment
``make check-avocado`` command and the ``AVOCADO_TAGS`` environment
variable:
.. code::
make check-acceptance AVOCADO_TAGS=quick
make check-avocado AVOCADO_TAGS=quick
Note that tags separated with commas have an AND behavior, while tags
separated by spaces have an OR behavior. For more information on Avocado
@ -728,31 +727,31 @@ tags, see:
https://avocado-framework.readthedocs.io/en/latest/guides/user/chapters/tags.html
To run a single test file, a couple of them, or a test within a file
using the ``make check-acceptance`` command, set the ``AVOCADO_TESTS``
using the ``make check-avocado`` command, set the ``AVOCADO_TESTS``
environment variable with the test files or test names. To run all
tests from a single file, use:
.. code::
make check-acceptance AVOCADO_TESTS=$FILEPATH
make check-avocado AVOCADO_TESTS=$FILEPATH
The same is valid to run tests from multiple test files:
.. code::
make check-acceptance AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
make check-avocado AVOCADO_TESTS='$FILEPATH1 $FILEPATH2'
To run a single test within a file, use:
.. code::
make check-acceptance AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
make check-avocado AVOCADO_TESTS=$FILEPATH:$TESTCLASS.$TESTNAME
The same is valid to run single tests from multiple test files:
.. code::
make check-acceptance AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
make check-avocado AVOCADO_TESTS='$FILEPATH1:$TESTCLASS1.$TESTNAME1 $FILEPATH2:$TESTCLASS2.$TESTNAME2'
The scripts installed inside the virtual environment may be used
without an "activation". For instance, the Avocado test runner
@ -760,9 +759,9 @@ may be invoked by running:
.. code::
tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/acceptance/
tests/venv/bin/avocado run $OPTION1 $OPTION2 tests/avocado/
Note that if ``make check-acceptance`` was not executed before, it is
Note that if ``make check-avocado`` was not executed before, it is
possible to create the Python virtual environment with the dependencies
needed running:
@ -775,20 +774,20 @@ a test file. To run tests from a single file within the build tree, use:
.. code::
tests/venv/bin/avocado run tests/acceptance/$TESTFILE
tests/venv/bin/avocado run tests/avocado/$TESTFILE
To run a single test within a test file, use:
.. code::
tests/venv/bin/avocado run tests/acceptance/$TESTFILE:$TESTCLASS.$TESTNAME
tests/venv/bin/avocado run tests/avocado/$TESTFILE:$TESTCLASS.$TESTNAME
Valid test names are visible in the output from any previous execution
of Avocado or ``make check-acceptance``, and can also be queried using:
of Avocado or ``make check-avocado``, and can also be queried using:
.. code::
tests/venv/bin/avocado list tests/acceptance
tests/venv/bin/avocado list tests/avocado
Manual Installation
~~~~~~~~~~~~~~~~~~~
@ -806,16 +805,16 @@ Alternatively, follow the instructions on this link:
Overview
~~~~~~~~
The ``tests/acceptance/avocado_qemu`` directory provides the
The ``tests/avocado/avocado_qemu`` directory provides the
``avocado_qemu`` Python module, containing the ``avocado_qemu.Test``
class. Here's a simple usage example:
.. code::
from avocado_qemu import Test
from avocado_qemu import QemuSystemTest
class Version(Test):
class Version(QemuSystemTest):
"""
:avocado: tags=quick
"""
@ -860,10 +859,10 @@ and hypothetical example follows:
.. code::
from avocado_qemu import Test
from avocado_qemu import QemuSystemTest
class MultipleMachines(Test):
class MultipleMachines(QemuSystemTest):
def test_multiple_machines(self):
first_machine = self.get_vm()
second_machine = self.get_vm()
@ -913,7 +912,7 @@ like this:
self.ssh_command('some_command_to_be_run_in_the_guest')
Please refer to tests that use ``avocado_qemu.LinuxTest`` under
``tests/acceptance`` for more examples.
``tests/avocado`` for more examples.
QEMUMachine
~~~~~~~~~~~
@ -1204,7 +1203,7 @@ And remove any package you want with::
pip uninstall <package_name>
If you've used ``make check-acceptance``, the Python virtual environment where
If you've used ``make check-avocado``, the Python virtual environment where
Avocado is installed will be cleaned up as part of ``make check-clean``.
.. _checktcg-ref: