Commit graph

60 commits

Author SHA1 Message Date
Thomas Huth
9fa4fc23e3 tests/functional: Add a helper function for retrieving the hostfwd port
It's just a wrapper around get_info_usernet_hostfwd_port from the
qemu module that is also calling the right monitor command for
retrieving the information from QEMU.

Message-ID: <20241217121550.141072-2-thuth@redhat.com>
Acked-by: Michael S. Tsirkin <mst@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 20:33:29 +01:00
Daniel P. Berrangé
674a750b51 tests/functional: ignore errors when caching assets, except for 404
We see periodic errors caching assets due to a combination of transient
networking and server problems. With the previous patch to skip running
a test when it has missing assets, we can now treat most cache download
errors as non-fatal.

Only HTTP 404 is retained as fatal, since it is a strong indicator of
a fully broken test rather than a transient error.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-32-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
6ff217c2d1 tests/functional: skip tests if assets are not available
If downloading of assets has been disabled, then skip running a
test if the assets it has registered are not already downloaded.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-31-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
c5be9dd32f tests/functional: remove now unused 'run_cmd' helper
All usage has been replaced by direct 'subprocess' helpers.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-30-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
37e9b19c34 tests/functional: replace 'run_cmd' with subprocess helpers
The 'run_cmd' helper is re-implementing a convenient helper that
already exists in the form of the 'run' and 'check_call' methods
provided by 'subprocess'.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-29-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
3bb4c8b613 tests/functional: drop back compat imports from utils.py
Now that all tests are converted over to the higher level wrapper
functions, the back compat imports from utils.py are redundant.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-28-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
fd4abcb008 tests/functional: add 'uncompress' to QemuBaseTest
This helper wrappers utils.uncompress, forcing the use of the scratch
directory, to ensure any uncompressed files are cleaned at test
termination.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-26-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
dd66e65f05 tests/functional: add a generalized uncompress helper
There are many types of compression that the tests deal with, and
it makes sense to have a single helper 'uncompress' that can deal
with all.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-25-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
5831ed84e7 tests/functional: convert tests to new archive_extract helper
Replace use of utils.archive_extract and extract_from_deb with the
new archive_extract helper.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-24-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
239fd29d6f tests/functional: add 'archive_extract' to QemuBaseTest
This helper wrappers archive.archive_extract, forcing the use of the
scratch directory, to ensure any extracted files are cleaned at test
termination. If a specific member is requested, then the path to the
extracted file is also returned.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-23-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
c283afbf65 tests/functional: add a generalized archive_extract
There are many types of archives that the tests deal with. Provide
a generalized 'archive_extract' that can detect the format and
delegate to the appropriate helper for extraction. This ensures
that all archive extraction code follows the same design pattern.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-22-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
c055f1d26f tests/functional: let cpio_extract accept filenames
Currently cpio_extract differs from tar_extract/zip_extract
in that it only allows a file-like object as input. Adapt it
to also support filenames.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-21-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
512fe088b1 tests/functional: add common deb_extract helper
This mirrors the existing archive_extract, cpio_extract and zip_extract
helpers

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-20-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
379ee839f9 tests/functional: add common zip_extract helper
This mirrors the existing archive_extract and cpio_extract helpers

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-19-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
ba32e50a1b tests/functional: move uncompress handling into new uncompress.py file
More uncompress related code will be added shortly, so having a
separate file makes more sense.

The utils.py imports the functions from archive.py, so that
existing callers don't need to be modified. This avoids
redundant code churn until later in the series when all
calls will be adapted for other reasons.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-18-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
cfcb4484fc tests/functional: move archive handling into new archive.py file
More archive related code will be added shortly, so having a
separate file makes more sense.

The utils.py imports the functions from archive.py, so that
existing callers don't need to be modified. This avoids
redundant code churn until later in the series when all
calls will be adapted for other reasons.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-17-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
beaf88c895 tests/functional: switch over to using self.scratch_file()
Replace any instances of

  os.path.join(self.workdir, ".../...")
  self.workdir + "/.../..."

with

  self.scratch_file("...", "...")

which is more compact and portable

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-15-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
8b5a0dd3a8 tests/functional: switch over to using self.build_file(...)
This removes direct access of the 'BUILD_DIR' variable.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-13-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
bcc12768c2 tests/functional: switch over to using self.log_file(...)
This removes direct access of the 'self.logdir' variable.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-12-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
f84f8e71eb tests/functional: add helpers for building file paths
Add helper methods that construct paths for

 * log files - to be preserved at the end of a test
 * scratch files - to be purged at the end of a test
 * build files - anything relative to the build root
 * data files - anything relative to the functional test source root
 * socket files - a short temporary dir to avoid UNIX socket limits

These are to be used instead of direct access to the self.workdir,
or self.logdir variables, or any other place where paths are built
manually.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-11-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
9132fff802 tests/functional: drop 'has_cmd' and 'has_cmds' helpers
The 'which' helper is simpler, not depending on the external 'which'
binary, and is sufficient for test needs.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-10-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
3ea06d65be tests/functional: introduce some helpful decorators
Reduce repeated boilerplate with some helper decorators:

 @skipIfNotPlatform("x86_64", "aarch64")

  => Skip unless the build host platform matches

 @skipIfMissingCommands("mkisofs", "losetup")

  => Skips unless all listed commands are found in $PATH

 @skipIfMissingImports("numpy", "cv2")

  => Skips unless all listed modules can be imported

 @skipFlakyTest("https://gitlab.com/qemu-project/qemu/-/issues/NNN")

  => Skips unless env var requests flaky tests with the
     reason documented in the referenced gitlab bug

 @skipBigData

  => Skips unless env var permits tests creating big data files

 @skipUntrustedTest

  => Skips unless env var permits tests which are potentially
     dangerous to the host

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-8-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
c54edc7109 tests/functional: drop 'tesseract_available' helper
Platforms we target have new enough tesseract that it suffices to merely
check if the binary exists.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20241217155953.3950506-7-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
dd6dfc012c tests/functional: simplify 'which' implementation
The 'access' check implies the file exists.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-6-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
0da341a78f tests/functional: remove duplicated 'which' function impl
Put the 'which' function into shared code.

Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-5-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
c27f452d61 tests/functional: resolve str(Asset) to cache file path
Allow an Asset object to be used in place of a filename but
making its string representation resolve to the cache file
path.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20241217155953.3950506-4-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
8a6253a43a tests/functional: remove many unused imports
Identified using 'pylint --disable=all --enable=W0611'

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20241217155953.3950506-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-12-17 19:39:53 +01:00
Daniel P. Berrangé
f5578e427f tests/functional: avoid accessing log_filename on earlier failures
If a failure occurs early in the QemuBaseTest constructor, the
'log_filename' object atttribute may not exist yet. This happens
most notably if the QEMU_TEST_QEMU_BINARY is not set. We can't
initialize 'log_filename' earlier as we use the binary to identify
the architecture which is then used to build the path in which the
logs are stored.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-19-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-19-alex.bennee@linaro.org>
2024-11-25 10:13:42 +00:00
Daniel P. Berrangé
1a8755a51e tests/functional: add a QMP backdoor for debugging stalled tests
Support the QEMU_TEST_QMP_BACKDOOR=backdoor.sock env variable as a
way to get a QMP backdoor for debugging a stalled QEMU test. Most
typically this would be used if running the tests directly:

 $ QEMU_TEST_QMP_BACKDOOR=backdoor.sock \
   QEMU_TEST_QEMU_BINARY=./build/qemu-system-arm \
   PYTHONPATH=./python \
   ./tests/functional/test_arm_tuxrun.py

And then, when the test stalls, in a second shell run:

 $ ./scripts/qmp/qmp-shell backdoor.sock

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-18-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-18-alex.bennee@linaro.org>
2024-11-25 10:13:41 +00:00
Daniel P. Berrangé
97d79319f0 tests/functional: remove time.sleep usage from tuxrun tests
The tuxrun tests send a series of strings to the guest to login
and then run commands. Since we have been unable to match on
console output that isn't followed by a newline, the test used
many time.sleep() statements to pretend to synchronize with
the guest.

This has proved to be unreliable for the aarch64be instance of
the tuxrun tests, with the test often hanging. The hang is a
very subtle timing problem, and it is suspected that some
(otherwise apparently harmless) I/O error messages could be
resulting in full FIFO buffers, stalling interaction with
the guest.

With the newly rewritten console interaction able to match
strings that don't have a following newline, the tux run
tests can now match directly on the login prompt, and/or
shell PS1 prompt.

Resolves: https://gitlab.com/qemu-project/qemu/-/issues/2689
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-17-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-17-alex.bennee@linaro.org>
2024-11-25 10:13:41 +00:00
Daniel P. Berrangé
cdad03b74f tests/functional: rewrite console handling to be bytewise
The console interaction that waits for predicted strings uses
readline(), and thus is only capable of waiting for strings
that are followed by a newline.

This is inconvenient when needing to match on some things,
particularly login prompts, or shell prompts, causing tests
to use time.sleep(...) instead, which is unreliable.

Switch to reading the console 1 byte at a time, comparing
against the success/failure messages until we see a match,
regardless of whether a newline is encountered.

The success/failure comparisons are done with the python bytes
type, rather than strings, to avoid the problem of needing to
decode partially received multibyte utf8 characters.

Heavily inspired by a patch proposed by Cédric, but written
again to work in bytes, rather than strings.

Co-developed-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-16-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-16-alex.bennee@linaro.org>
2024-11-25 10:13:41 +00:00
Daniel P. Berrangé
f03a81897d tests/functional: require non-NULL success_message for console wait
When waiting for expected output, the 'success_message' is a mandatory
parameter, with 'failure_message' defaulting to None.

The code has logic which indicates it was trying to cope with
'success_message' being None and 'failure_message' being non-None but
it does not appear able to actually do anything useful. The check for
'success_message is None' will break out of the loop before any check
for 'failure_message' has been performed.

IOW, for practcal purposes 'success_message' must be non-None unless
'send_string' is set. Assert this expectation and simplify the loop
logic.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-15-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-15-alex.bennee@linaro.org>
2024-11-25 10:13:41 +00:00
Daniel P. Berrangé
6f0942b723 tests/functional: logs details of console interaction operations
When functional tests go wrong, it will often be related to the console
interaction wait state. By logging the messages that we're looking for,
and data we're about to be sending, it'll be easier to diagnose where
tests are getting stuck.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-13-berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-13-alex.bennee@linaro.org>
2024-11-25 10:13:41 +00:00
Daniel P. Berrangé
9bcfead1ee tests/functional: enable debug logging for QEMUMachine
Set the 'qemu.machine' logger to 'DEBUG' level, to ensure we see log
messages related to the QEMUMachine class. Most importantly this
ensures we capture the full QEMU command line args for instances we
spawn.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-12-berrange@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-Id: <20241121165806.476008-12-alex.bennee@linaro.org>
2024-11-25 10:13:33 +00:00
Daniel P. Berrangé
9f85aff93f tests/functional: honour requested test VM name in QEMUMachine
The functional test case class is going to the trouble of passing
around a machine name, but then fails to give this QEMUMachine. As
a result, QEMUMachine will create a completely random name. Since
log file names match the machine name, this results in log files
accumulating over time.

Reviewed-by: Thomas Huth <thuth@redhat.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-11-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-11-alex.bennee@linaro.org>
2024-11-25 10:06:20 +00:00
Daniel P. Berrangé
e6d69e0f3b tests/functional: put QEMUMachine logs in testcase log directory
We are not passing the 'log_dir' parameter to QEMUMachine, so the
QEMU stdout/err logs are being placed in a temp directory and thus
deleted after execution. This makes them inaccessible as gitlab
CI artifacts.

Pass the testcase log directory path into QEMUMachine to make the
logs persistent.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-10-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-10-alex.bennee@linaro.org>
2024-11-25 10:06:20 +00:00
Daniel P. Berrangé
57e504ad4f tests/functional: remove obsolete reference to avocado bug
Historical bugs in avocado related to zstd support are not relevant to
the code now that it uses QEMU's native test harness.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-7-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-7-alex.bennee@linaro.org>
2024-11-25 10:06:20 +00:00
Daniel P. Berrangé
dbaaef7dbf tests/functional: automatically clean up scratch files after tests
The build/tests/functional subdirectories are consuming huge amounts
of disk space.

Split the location for scratch files into a 'scratch' sub-directory,
separate from log files, and delete it upon completion of each test.
The new env variable QEMU_TEST_KEEP_SCRATCH can be set to preserve
this scratch dir for debugging access if required.

Reviewed-by: Alex Bennée <alex.bennee@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-Id: <20241121154218.1423005-3-berrange@redhat.com>
Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Message-Id: <20241121165806.476008-3-alex.bennee@linaro.org>
2024-11-25 10:06:20 +00:00
Thomas Huth
bb986e4068 tests/functional: Provide the user with hints where to find more log files
Since the base.log and console.log files are not referenced from the
meson test logs yet, they might be hard to find for the casual users.
Thus let's print some hints in case a test case failed. For this we
have to run unittest.main() with exit=False to get the results of the
testing. Then we can iterate through the failed test cases to print
out the information accordingly.

Message-ID: <20241105123849.359391-1-thuth@redhat.com>
Reviewed-by: Cédric Le Goater <clg@redhat.com>
Tested-by: Cédric Le Goater <clg@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-11-08 06:16:38 +01:00
Thomas Huth
f7d6b77220 tests/functional: Convert BananaPi tests to the functional framework
Move the BananaPi tests from tests/avocado/boot_linux_console.py into
a new file dedicated for Banana Pi tests in the functional framework.
Update the hash sums of the assets to sha256 along the way and fix the
broken link for the buildroot image from storage.kernelci.org.

(Note: The test_arm_bpim2u_openwrt_22_03_3 test is currently broken
due to a regression in commit 4c2c047469 ("target/arm: Fix usage of MMU
indexes when EL3 is AArch32") - it works if that commit gets reverted)

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20241029092440.25021-2-thuth@redhat.com>
2024-11-04 14:16:12 +01:00
Daniel P. Berrangé
786bc22552 tests/functional: make cached asset files read-only
This ensures that if a functional test runs QEMU with a writable
disk pointing to a cached asset, an error will be reported, rather
than silently modifying the cache file.

As an example, tweaking test_sbsaref.py to set snapshot=off,
results in a clear error:

  Command: ./build/qemu-system-aarch64 ...snip... -drive file=/var/home/berrange/.cache/qemu/download/44cdbae275ef1bb6dab1d5fbb59473d4f741e1c8ea8a80fd9e906b531d6ad461,format=raw,snapshot=off -cpu max,pauth=off
  Output: qemu-system-aarch64: Could not open '/var/home/berrange/.cache/qemu/download/44cdbae275ef1bb6dab1d5fbb59473d4f741e1c8ea8a80fd9e906b531d6ad461': Permission denied

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Tested-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20241025092659.2312118-3-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-11-04 14:16:11 +01:00
Daniel P. Berrangé
c9daf680d1 tests/functional: make tuxrun disk images writable
The zstd command will preserve the input archive permissions on the
output file. So when we decompress the readonly cached image, the
resulting per-test run private disk image will also be readonly.
We need it to be writable, so make it so.

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20241025092659.2312118-2-berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-11-04 14:16:11 +01:00
Thomas Huth
c1b24f0fb7 tests/functional: Add a base class for the TuxRun tests
Add a base class for the TuxRun tests, based on the code from
tests/avocado/tuxrun_baselines.py (the test have to be put into
separate file in the following commits, depending on the target
architecture that gets tested).

Signed-off-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20241011131937.377223-2-thuth@redhat.com>
2024-10-21 13:25:12 +02:00
Thomas Huth
db17daf8c4 tests/functional: Fix hash validation
The _check() function is supposed to check whether the hash of the
downloaded file matches the expected one. Unfortunately, during the
last rework of this function, the check was accidentally turned into
returning the hash value itself instead of a True/False value,
effectively accepting each hash as valid. Let's do a proper check
again now.

Fixes:05e303210d ("tests/functional/qemu_test: Use Python hashlib ...")
Signed-off-by: Thomas Huth <thuth@redhat.com>
Reviewed-by: Michael Tokarev <mjt@tls.msk.ru>
Signed-off-by: Michael Tokarev <mjt@tls.msk.ru>
2024-10-04 18:53:53 +03:00
Thomas Huth
d2a500cea9 tests/functional/qemu_test: Add a function for launching kernels more easily
The task for launching a kernel is quite repetitive: Set the serial
console, set the -kernel and maybe -initrd and -dtb parameters,
launch the VM and then wait for the expected console output. So
it's easier in some tests to provide these steps via a separate
function.

Message-ID: <20240919185749.71222-2-thuth@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-24 13:38:29 +02:00
Thomas Huth
05e303210d tests/functional/qemu_test: Use Python hashlib instead of external programs
Some systems (like OpenBSD) do not have the sha256sum or sha512sum programs
installed by default, or use different names for those. Use the Python
hashlib instead so we don't have to rely on the external programs.

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Message-ID: <20240910201742.239559-1-thuth@redhat.com>
Reviewed-by: Brian Cain <bcain@quicinc.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-11 09:49:12 +02:00
Thomas Huth
59d100243d tests/functional: Fix bad usage of has_cmd
has_cmd returns a tuple, not a boolean value. This fixes a crash when
e.g. "tesseract" is not available in the test_m68k_nextcube test.

Reported-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20240910075820.51346-1-thuth@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-11 09:49:12 +02:00
Thomas Huth
8720471efd tests/functional: Add the LinuxKernelTest for testing the Linux boot process
Copy the LinuxKernelTest from tests/acceptance/boot_linux_console.py
to be able to convert the related tests to the functional test framework
in the following patches.

Message-ID: <20240906180549.792832-2-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-11 09:49:11 +02:00
Philippe Mathieu-Daudé
34917ead72 tests/functional: Convert ARM bFLT linux-user avocado test
Straight forward conversion. Update the SHA1 hashes to
SHA256 hashes since SHA1 should not be used anymore nowadays.
Expose cpio_extract() in qemu_test.utils for possible reuse.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240822104238.75045-3-philmd@linaro.org>
[thuth: Add test to meson.build]
Message-ID: <20240830133841.142644-39-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-04 12:28:00 +02:00
Philippe Mathieu-Daudé
99465d3fe4 tests/functional: Add QemuUserTest class
Per commit 5334df4822 ("tests/avocado: Introduce
QemuUserTest base class"):

  Similarly to the 'System' Test base class with methods
  for testing system emulation, the QemuUserTest class
  contains methods useful to test user-mode emulation.

Signed-off-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Message-ID: <20240822104238.75045-2-philmd@linaro.org>
Reviewed-by: Thomas Huth <thuth@redhat.com>
Message-ID: <20240830133841.142644-38-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2024-09-04 12:28:00 +02:00