motorhead/qemu
1
0
Fork 0
mirror of https://github.com/Motorhead1991/qemu.git synced 2026-01-05 22:17:40 -07:00
Code Issues Projects Releases Packages Wiki Activity Actions

tests/functional: enable pre-emptive caching of assets

Browse source 
Many tests need to access assets stored on remote sites. We don't want
to download these during test execution when run by meson, since this
risks hitting test timeouts when data transfers are slow.

Add support for pre-emptive caching of assets by setting the env var
QEMU_TEST_PRECACHE to point to a timestamp file. When this is set,
instead of running the test, the assets will be downloaded and saved
to the cache, then the timestamp file created.

A meson custom target is created as a dependency of each test suite
to trigger the pre-emptive caching logic before the test runs.

When run in caching mode, it will locate assets by looking for class
level variables with a name prefix "ASSET_", and type "Asset".

At the ninja level

   ninja test --suite functional

will speculatively download any assets that are not already cached,
so it is advisable to set a timeout multiplier.

   QEMU_TEST_NO_DOWNLOAD=1 ninja test --suite functional

will fail the test if a required asset is not already cached

   ninja precache-functional

will download and cache all assets required by the functional
tests

At the make level, precaching is always done by

   make check-functional

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Tested-by: Richard Henderson <richard.henderson@linaro.org>
[thuth: Remove the duplicated "path = os.path.basename(...)" line]
Message-ID: <20240830133841.142644-16-thuth@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
This commit is contained in:
Daniel P. Berrangé 2024-08-30 15:38:09 +02:00 • committed by Thomas Huth
parent 9903217a4e
commit f57213f85b
4 changed files with 74 additions and 3 deletions
Download patch file Download diff file Expand all files Collapse all files

34
tests/functional/qemu_test/asset.py
View file

@ -9,6 +9,8 @@ import hashlib
import logging
import os
import subprocess
import sys
import unittest
import urllib.request
from pathlib import Path
from shutil import copyfileobj
@ -62,6 +64,9 @@ class Asset:
self.cache_file, self.url)
return str(self.cache_file)
if os.environ.get("QEMU_TEST_NO_DOWNLOAD", False):
raise Exception("Asset cache is invalid and downloads disabled")
self.log.info("Downloading %s to %s...", self.url, self.cache_file)
tmp_cache_file = self.cache_file.with_suffix(".download")
@ -95,3 +100,32 @@ class Asset:
self.log.info("Cached %s at %s" % (self.url, self.cache_file))
return str(self.cache_file)
def precache_test(test):
log = logging.getLogger('qemu-test')
log.setLevel(logging.DEBUG)
handler = logging.StreamHandler(sys.stdout)
handler.setLevel(logging.DEBUG)
formatter = logging.Formatter(
'%(asctime)s - %(name)s - %(levelname)s - %(message)s')
handler.setFormatter(formatter)
log.addHandler(handler)
for name, asset in vars(test.__class__).items():
if name.startswith("ASSET_") and type(asset) == Asset:
log.info("Attempting to cache '%s'" % asset)
asset.fetch()
log.removeHandler(handler)
def precache_suite(suite):
for test in suite:
if isinstance(test, unittest.TestSuite):
Asset.precache_suite(test)
elif isinstance(test, unittest.TestCase):
Asset.precache_test(test)
def precache_suites(path, cacheTstamp):
loader = unittest.loader.defaultTestLoader
tests = loader.loadTestsFromNames([path], None)
with open(cacheTstamp, "w") as fh:
Asset.precache_suite(tests)