tests/functional: Convert the acpi-bits test into a standalone test Mostly a straight-forward conversion. Looks like we can simply drop the avocado datadrainer stuff when not using the avocado framework anymore. Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org> Tested-by: Philippe Mathieu-Daudé <philmd@linaro.org> Message-ID: <20240830133841.142644-30-thuth@redhat.com> Signed-off-by: Thomas Huth <thuth@redhat.com>
diff --git a/MAINTAINERS b/MAINTAINERS index 70dd243..f234464 100644 --- a/MAINTAINERS +++ b/MAINTAINERS
@@ -2074,8 +2074,8 @@ M: Ani Sinha <anisinha@redhat.com> M: Michael S. Tsirkin <mst@redhat.com> S: Supported -F: tests/avocado/acpi-bits/* -F: tests/avocado/acpi-bits.py +F: tests/functional/acpi-bits/* +F: tests/functional/test_acpi_bits.py F: docs/devel/acpi-bits.rst ACPI/HEST/GHES
diff --git a/docs/devel/acpi-bits.rst b/docs/devel/acpi-bits.rst index 1ec394f..78aeb6a 100644 --- a/docs/devel/acpi-bits.rst +++ b/docs/devel/acpi-bits.rst
@@ -1,6 +1,6 @@ -============================================================================= -ACPI/SMBIOS avocado tests using biosbits -============================================================================= +================================== +ACPI/SMBIOS testing using biosbits +================================== ************ Introduction ************ @@ -35,7 +35,7 @@ For QEMU, we maintain a fork of bios bits in gitlab along with all the dependent submodules `here <https://gitlab.com/qemu-project/biosbits-bits>`__. This fork contains numerous fixes, a newer acpica and changes specific to -running this avocado QEMU tests using bits. The author of this document +running these functional QEMU tests using bits. The author of this document is the sole maintainer of the QEMU fork of bios bits repository. For more information, please see author's `FOSDEM talk on this bios-bits based test framework <https://fosdem.org/2024/schedule/event/fosdem-2024-2262-exercising-qemu-generated-acpi-smbios-tables-using-biosbits-from-within-a-guest-vm-/>`__. @@ -44,12 +44,12 @@ Description of the test framework ********************************* -Under the directory ``tests/avocado/``, ``acpi-bits.py`` is a QEMU avocado -test that drives all this. +Under the directory ``tests/functional/``, ``test_acpi_bits.py`` is a QEMU +functional test that drives all this. A brief description of the various test files follows. -Under ``tests/avocado/`` as the root we have: +Under ``tests/functional/`` as the root we have: :: @@ -60,12 +60,12 @@ │ ├── smbios.py2 │ ├── testacpi.py2 │ └── testcpuid.py2 - ├── acpi-bits.py + ├── test_acpi_bits.py -* ``tests/avocado``: +* ``tests/functional``: - ``acpi-bits.py``: - This is the main python avocado test script that generates a + ``test_acpi_bits.py``: + This is the main python functional test script that generates a biosbits iso. It then spawns a QEMU VM with it, collects the log and reports test failures. This is the script one would be interested in if they wanted to add or change some component of the log parsing, add a new command line @@ -79,35 +79,22 @@ you to inspect and run the specific commands manually. In order to run this test, please perform the following steps from the QEMU - build directory: + build directory (assuming that the sources are in ".."): :: - $ make check-venv (needed only the first time to create the venv) - $ ./pyvenv/bin/avocado run -t acpi tests/avocado + $ export PYTHONPATH=../python:../tests/functional + $ export QEMU_TEST_QEMU_BINARY=$PWD/qemu-system-x86_64 + $ python3 ../tests/functional/test_acpi_bits.py - The above will run all acpi avocado tests including this one. - In order to run the individual tests, perform the following: - :: + The above will run all acpi-bits functional tests (producing output in + tap format). - $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py --tap - + You can inspect the log files in tests/functional/x86_64/test_acpi_bits.*/ + for more information about the run or in order to diagnoze issues. + If you pass V=1 in the environment, more diagnostic logs will be put into + the test log. - The above will produce output in tap format. You can omit "--tap -" in the - end and it will produce output like the following: - :: - - $ ./pyvenv/bin/avocado run tests/avocado/acpi-bits.py - Fetching asset from tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits - JOB ID : eab225724da7b64c012c65705dc2fa14ab1defef - JOB LOG : /home/anisinha/avocado/job-results/job-2022-10-10T17.58-eab2257/job.log - (1/1) tests/avocado/acpi-bits.py:AcpiBitsTest.test_acpi_smbios_bits: PASS (33.09 s) - RESULTS : PASS 1 | ERROR 0 | FAIL 0 | SKIP 0 | WARN 0 | INTERRUPT 0 | CANCEL 0 - JOB TIME : 39.22 s - - You can inspect the log file for more information about the run or in order - to diagnoze issues. If you pass V=1 in the environment, more diagnostic logs - would be found in the test log. - -* ``tests/avocado/acpi-bits/bits-config``: +* ``tests/functional/acpi-bits/bits-config``: This location contains biosbits configuration files that determine how the software runs the tests. @@ -117,7 +104,7 @@ or actions are performed by bits. The description of the config options are provided in the file itself. -* ``tests/avocado/acpi-bits/bits-tests``: +* ``tests/functional/acpi-bits/bits-tests``: This directory contains biosbits python based tests that are run from within the biosbits environment in the spawned VM. New additions of test cases can @@ -155,7 +142,8 @@ (a) They are python2.7 based scripts and not python 3 scripts. (b) They are run from within the bios bits VM and is not subjected to QEMU build/test python script maintenance and dependency resolutions. - (c) They need not be loaded by avocado framework when running tests. + (c) They need not be loaded by the test framework by accident when running + tests. Author: Ani Sinha <anisinha@redhat.com>
diff --git a/tests/avocado/acpi-bits/bits-config/bits-cfg.txt b/tests/functional/acpi-bits/bits-config/bits-cfg.txt similarity index 100% rename from tests/avocado/acpi-bits/bits-config/bits-cfg.txt rename to tests/functional/acpi-bits/bits-config/bits-cfg.txt
diff --git a/tests/avocado/acpi-bits/bits-tests/smbios.py2 b/tests/functional/acpi-bits/bits-tests/smbios.py2 similarity index 100% rename from tests/avocado/acpi-bits/bits-tests/smbios.py2 rename to tests/functional/acpi-bits/bits-tests/smbios.py2
diff --git a/tests/avocado/acpi-bits/bits-tests/smilatency.py2 b/tests/functional/acpi-bits/bits-tests/smilatency.py2 similarity index 100% rename from tests/avocado/acpi-bits/bits-tests/smilatency.py2 rename to tests/functional/acpi-bits/bits-tests/smilatency.py2
diff --git a/tests/avocado/acpi-bits/bits-tests/testacpi.py2 b/tests/functional/acpi-bits/bits-tests/testacpi.py2 similarity index 100% rename from tests/avocado/acpi-bits/bits-tests/testacpi.py2 rename to tests/functional/acpi-bits/bits-tests/testacpi.py2
diff --git a/tests/avocado/acpi-bits/bits-tests/testcpuid.py2 b/tests/functional/acpi-bits/bits-tests/testcpuid.py2 similarity index 100% rename from tests/avocado/acpi-bits/bits-tests/testcpuid.py2 rename to tests/functional/acpi-bits/bits-tests/testcpuid.py2
diff --git a/tests/functional/meson.build b/tests/functional/meson.build index 61cdd1d..3bbe80b 100644 --- a/tests/functional/meson.build +++ b/tests/functional/meson.build
@@ -11,6 +11,7 @@ # Timeouts for individual tests that can be slow e.g. with debugging enabled test_timeouts = { + 'acpi_bits' : 240, 'netdev_ethtool' : 180, 'ppc_40p' : 240, 'ppc64_hv' : 1000, @@ -96,6 +97,7 @@ ] tests_x86_64_system_thorough = [ + 'acpi_bits', 'netdev_ethtool', 'virtio_gpu', ]
diff --git a/tests/avocado/acpi-bits.py b/tests/functional/test_acpi_bits.py old mode 100644 new mode 100755 similarity index 85% rename from tests/avocado/acpi-bits.py rename to tests/functional/test_acpi_bits.py index efe4f52..ee40647 --- a/tests/avocado/acpi-bits.py +++ b/tests/functional/test_acpi_bits.py
@@ -1,5 +1,5 @@ #!/usr/bin/env python3 -# group: rw quick +# # Exercise QEMU generated ACPI/SMBIOS tables using biosbits, # https://biosbits.org/ # @@ -24,7 +24,7 @@ # pylint: disable=consider-using-f-string """ -This is QEMU ACPI/SMBIOS avocado tests using biosbits. +This is QEMU ACPI/SMBIOS functional tests using biosbits. Biosbits is available originally at https://biosbits.org/. This test uses a fork of the upstream bits and has numerous fixes including an upgraded acpica. The fork is located here: @@ -41,15 +41,16 @@ import tempfile import time import zipfile + +from pathlib import Path from typing import ( List, Optional, Sequence, ) from qemu.machine import QEMUMachine -from avocado import skipIf -from avocado.utils import datadrainer as drainer -from avocado_qemu import QemuBaseTest +from unittest import skipIf +from qemu_test import QemuBaseTest, Asset deps = ["xorriso", "mformat"] # dependent tools needed in the test setup/box. supported_platforms = ['x86_64'] # supported test platforms. @@ -129,34 +130,32 @@ def base_args(self): class AcpiBitsTest(QemuBaseTest): #pylint: disable=too-many-instance-attributes """ ACPI and SMBIOS tests using biosbits. - - :avocado: tags=arch:x86_64 - :avocado: tags=acpi - """ # in slower systems the test can take as long as 3 minutes to complete. timeout = BITS_TIMEOUT + # following are some standard configuration constants + # gitlab CI does shallow clones of depth 20 + BITS_INTERNAL_VER = 2020 + # commit hash must match the artifact tag below + BITS_COMMIT_HASH = 'c7920d2b' + # this is the latest bits release as of today. + BITS_TAG = "qemu-bits-10262023" + + ASSET_BITS = Asset(("https://gitlab.com/qemu-project/" + "biosbits-bits/-/jobs/artifacts/%s/" + "download?job=qemu-bits-build" % BITS_TAG), + '1b8dd612c6831a6b491716a77acc486666aaa867051cdc34f7ce169c2e25f487') + def __init__(self, *args, **kwargs): super().__init__(*args, **kwargs) self._vm = None self._workDir = None self._baseDir = None - # following are some standard configuration constants - self._bitsInternalVer = 2020 # gitlab CI does shallow clones of depth 20 - self._bitsCommitHash = 'c7920d2b' # commit hash must match - # the artifact tag below - self._bitsTag = "qemu-bits-10262023" # this is the latest bits - # release as of today. - self._bitsArtSHA1Hash = 'b22cdfcfc7453875297d06d626f5474ee36a343f' - self._bitsArtURL = ("https://gitlab.com/qemu-project/" - "biosbits-bits/-/jobs/artifacts/%s/" - "download?job=qemu-bits-build" %self._bitsTag) self._debugcon_addr = '0x403' self._debugcon_log = 'debugcon-log.txt' - logging.basicConfig(level=logging.INFO) - self.logger = logging.getLogger('acpi-bits') + self.logger = self.log def _print_log(self, log): self.logger.info('\nlogs from biosbits follows:') @@ -171,7 +170,7 @@ def copy_bits_config(self): bits_config_dir = os.path.join(self._baseDir, 'acpi-bits', 'bits-config') target_config_dir = os.path.join(self._workDir, - 'bits-%d' %self._bitsInternalVer, + 'bits-%d' %self.BITS_INTERNAL_VER, 'boot') self.assertTrue(os.path.exists(bits_config_dir)) self.assertTrue(os.path.exists(target_config_dir)) @@ -188,7 +187,7 @@ def copy_test_scripts(self): bits_test_dir = os.path.join(self._baseDir, 'acpi-bits', 'bits-tests') target_test_dir = os.path.join(self._workDir, - 'bits-%d' %self._bitsInternalVer, + 'bits-%d' %self.BITS_INTERNAL_VER, 'boot', 'python') self.assertTrue(os.path.exists(bits_test_dir)) @@ -248,9 +247,9 @@ def generate_bits_iso(self): test scripts """ bits_dir = os.path.join(self._workDir, - 'bits-%d' %self._bitsInternalVer) + 'bits-%d' %self.BITS_INTERNAL_VER) iso_file = os.path.join(self._workDir, - 'bits-%d.iso' %self._bitsInternalVer) + 'bits-%d.iso' %self.BITS_INTERNAL_VER) mkrescue_script = os.path.join(self._workDir, 'grub-inst-x86_64-efi', 'bin', 'grub-mkrescue') @@ -264,8 +263,12 @@ def generate_bits_iso(self): try: if os.getenv('V') or os.getenv('BITS_DEBUG'): - subprocess.check_call([mkrescue_script, '-o', iso_file, - bits_dir], stderr=subprocess.STDOUT) + proc = subprocess.run([mkrescue_script, '-o', iso_file, + bits_dir], + stdout=subprocess.PIPE, + stderr=subprocess.STDOUT, + check=True) + self.logger.info("grub-mkrescue output %s" % proc.stdout) else: subprocess.check_call([mkrescue_script, '-o', iso_file, bits_dir], @@ -282,8 +285,9 @@ def generate_bits_iso(self): def setUp(self): # pylint: disable=arguments-differ super().setUp('qemu-system-') + self.logger = self.log - self._baseDir = os.getenv('AVOCADO_TEST_BASEDIR') + self._baseDir = Path(__file__).parent # workdir could also be avocado's own workdir in self.workdir. # At present, I prefer to maintain my own temporary working @@ -300,15 +304,14 @@ def setUp(self): # pylint: disable=arguments-differ os.mkdir(prebuiltDir, mode=0o775) bits_zip_file = os.path.join(prebuiltDir, 'bits-%d-%s.zip' - %(self._bitsInternalVer, - self._bitsCommitHash)) + %(self.BITS_INTERNAL_VER, + self.BITS_COMMIT_HASH)) grub_tar_file = os.path.join(prebuiltDir, 'bits-%d-%s-grub.tar.gz' - %(self._bitsInternalVer, - self._bitsCommitHash)) + %(self.BITS_INTERNAL_VER, + self.BITS_COMMIT_HASH)) - bitsLocalArtLoc = self.fetch_asset(self._bitsArtURL, - asset_hash=self._bitsArtSHA1Hash) + bitsLocalArtLoc = self.ASSET_BITS.fetch() self.logger.info("downloaded bits artifacts to %s", bitsLocalArtLoc) # extract the bits artifact in the temp working directory @@ -369,7 +372,7 @@ def test_acpi_smbios_bits(self): """The main test case implementation.""" iso_file = os.path.join(self._workDir, - 'bits-%d.iso' %self._bitsInternalVer) + 'bits-%d.iso' %self.BITS_INTERNAL_VER) self.assertTrue(os.access(iso_file, os.R_OK)) @@ -393,12 +396,6 @@ def test_acpi_smbios_bits(self): self._vm.set_console() self._vm.launch() - self.logger.debug("Console output from bits VM follows ...") - c_drainer = drainer.LineLogger(self._vm.console_socket.fileno(), - logger=self.logger.getChild("console"), - stop_check=(lambda : - not self._vm.is_running())) - c_drainer.start() # biosbits has been configured to run all the specified test suites # in batch mode and then automatically initiate a vm shutdown. @@ -406,4 +403,8 @@ def test_acpi_smbios_bits(self): # with the avocado test timeout. self._vm.event_wait('SHUTDOWN', timeout=BITS_TIMEOUT) self._vm.wait(timeout=None) + self.logger.debug("Checking console output ...") self.parse_log() + +if __name__ == '__main__': + QemuBaseTest.main()