blob: 09725e8ea92634854f8835d6b43a9a97347f5a25 [file] [log] [blame]
Alex Bennée7266ecc2022-05-27 16:36:03 +01001.. _testing:
2
Fam Zheng4eb99562018-02-01 10:20:46 +08003Testing in QEMU
4===============
5
Alex Bennéec8d6e042024-07-29 15:44:06 +01006QEMU's testing infrastructure is fairly complex as it covers
7everything from unit testing and exercising specific sub-systems all
8the way to full blown acceptance tests. To get an overview of the
9tests you can run ``make check-help`` from either the source or build
10tree.
11
12Most (but not all) tests are also integrated into the meson build
13system so can be run directly from the build tree, for example:
14
15.. code::
16
17 [./pyvenv/bin/]meson test --suite qemu:softfloat
18
19will run just the softfloat tests.
20
21The rest of this document will cover the details for specific test
22groups.
Fam Zheng4eb99562018-02-01 10:20:46 +080023
24Testing with "make check"
Paolo Bonzini16e79e12021-09-07 17:21:58 +020025-------------------------
Fam Zheng4eb99562018-02-01 10:20:46 +080026
Alex Bennéec8d6e042024-07-29 15:44:06 +010027The "make check" testing family includes most of the C based tests in QEMU.
Fam Zheng4eb99562018-02-01 10:20:46 +080028
29The usual way to run these tests is:
30
31.. code::
32
33 make check
34
Thomas Huth316082b2020-01-22 14:40:20 +010035which includes QAPI schema tests, unit tests, QTests and some iotests.
36Different sub-types of "make check" tests will be explained below.
Fam Zheng4eb99562018-02-01 10:20:46 +080037
38Before running tests, it is best to build QEMU programs first. Some tests
39expect the executables to exist and will fail with obscure messages if they
40cannot find them.
41
42Unit tests
Paolo Bonzini16e79e12021-09-07 17:21:58 +020043~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +080044
45Unit tests, which can be invoked with ``make check-unit``, are simple C tests
46that typically link to individual QEMU object files and exercise them by
47calling exported functions.
48
49If you are writing new code in QEMU, consider adding a unit test, especially
50for utility modules that are relatively stateless or have few dependencies. To
51add a new unit test:
52
Wainer dos Santos Moschetta8db5c3e2021-03-18 14:44:07 -0300531. Create a new source file. For example, ``tests/unit/foo-test.c``.
Fam Zheng4eb99562018-02-01 10:20:46 +080054
552. Write the test. Normally you would include the header file which exports
56 the module API, then verify the interface behaves as expected from your
57 test. The test code should be organized with the glib testing framework.
58 Copying and modifying an existing test is usually a good idea.
59
Wainer dos Santos Moschetta8db5c3e2021-03-18 14:44:07 -0300603. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a
Paolo Bonzinibab88ea2020-10-06 08:49:55 -040061 dictionary called ``tests``. The values are any additional sources and
62 dependencies to be linked with the test. For a simple test whose source
Wainer dos Santos Moschetta8db5c3e2021-03-18 14:44:07 -030063 is in ``tests/unit/foo-test.c``, it is enough to add an entry like::
Fam Zheng4eb99562018-02-01 10:20:46 +080064
Paolo Bonzinibab88ea2020-10-06 08:49:55 -040065 {
66 ...
67 'foo-test': [],
68 ...
69 }
Fam Zheng4eb99562018-02-01 10:20:46 +080070
71Since unit tests don't require environment variables, the simplest way to debug
72a unit test failure is often directly invoking it or even running it under
73``gdb``. However there can still be differences in behavior between ``make``
74invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment
75variable (which affects memory reclamation and catches invalid pointers better)
76and gtester options. If necessary, you can run
77
78.. code::
Cleber Rosa92970812018-10-04 12:18:47 -040079
Fam Zheng4eb99562018-02-01 10:20:46 +080080 make check-unit V=1
81
82and copy the actual command line which executes the unit test, then run
83it from the command line.
84
85QTest
Paolo Bonzini16e79e12021-09-07 17:21:58 +020086~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +080087
88QTest is a device emulation testing framework. It can be very useful to test
89device models; it could also control certain aspects of QEMU (such as virtual
Eduardo Habkosta738a502020-10-05 16:52:26 -040090clock stepping), with a special purpose "qtest" protocol. Refer to
91:doc:`qtest` for more details.
Fam Zheng4eb99562018-02-01 10:20:46 +080092
93QTest cases can be executed with
94
95.. code::
96
97 make check-qtest
98
Bin Meng0b49bc12022-09-27 19:06:32 +080099Writing portable test cases
100~~~~~~~~~~~~~~~~~~~~~~~~~~~
101Both unit tests and qtests can run on POSIX hosts as well as Windows hosts.
102Care must be taken when writing portable test cases that can be built and run
103successfully on various hosts. The following list shows some best practices:
104
105* Use portable APIs from glib whenever necessary, e.g.: g_setenv(),
106 g_mkdtemp(), g_mkdir().
107* Avoid using hardcoded /tmp for temporary file directory.
108 Use g_get_tmp_dir() instead.
109* Bear in mind that Windows has different special string representation for
110 stdin/stdout/stderr and null devices. For example if your test case uses
111 "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on
112 Windows instead. Also IO redirection does not work on Windows, so avoid
113 using "2>nul" whenever necessary.
114* If your test cases uses the blkdebug feature, use relative path to pass
115 the config and image file paths in the command line as Windows absolute
116 path contains the delimiter ":" which will confuse the blkdebug parser.
Stefan Weil1e458f12022-10-30 11:59:44 +0100117* Use double quotes in your extra QEMU command line in your test cases
Bin Meng0b49bc12022-09-27 19:06:32 +0800118 instead of single quotes, as Windows does not drop single quotes when
119 passing the command line to QEMU.
120* Windows opens a file in text mode by default, while a POSIX compliant
121 implementation treats text files and binary files the same. So if your
122 test cases opens a file to write some data and later wants to compare the
123 written data with the original one, be sure to pass the letter 'b' as
124 part of the mode string to fopen(), or O_BINARY flag for the open() call.
125* If a certain test case can only run on POSIX or Linux hosts, use a proper
126 #ifdef in the codes. If the whole test suite cannot run on Windows, disable
127 the build in the meson.build file.
128
Fam Zheng4eb99562018-02-01 10:20:46 +0800129QAPI schema tests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200130~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800131
132The QAPI schema tests validate the QAPI parser used by QMP, by feeding
133predefined input to the parser and comparing the result with the reference
134output.
135
136The input/output data is managed under the ``tests/qapi-schema`` directory.
137Each test case includes four files that have a common base name:
138
139 * ``${casename}.json`` - the file contains the JSON input for feeding the
140 parser
141 * ``${casename}.out`` - the file contains the expected stdout from the parser
142 * ``${casename}.err`` - the file contains the expected stderr from the parser
143 * ``${casename}.exit`` - the expected error code
144
145Consider adding a new QAPI schema test when you are making a change on the QAPI
146parser (either fixing a bug or extending/modifying the syntax). To do this:
147
1481. Add four files for the new case as explained above. For example:
149
150 ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``.
151
1522. Add the new test in ``tests/Makefile.include``. For example:
153
154 ``qapi-schema += foo.json``
155
156check-block
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200157~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800158
Thomas Huth316082b2020-01-22 14:40:20 +0100159``make check-block`` runs a subset of the block layer iotests (the tests that
Vladimir Sementsov-Ogievskiyb25a9482021-01-25 21:50:52 +0300160are in the "auto" group).
Thomas Huth316082b2020-01-22 14:40:20 +0100161See the "QEMU iotests" section below for more information.
Fam Zheng4eb99562018-02-01 10:20:46 +0800162
Fam Zheng4eb99562018-02-01 10:20:46 +0800163QEMU iotests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200164------------
Fam Zheng4eb99562018-02-01 10:20:46 +0800165
166QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing
167framework widely used to test block layer related features. It is higher level
168than "make check" tests and 99% of the code is written in bash or Python
169scripts. The testing success criteria is golden output comparison, and the
170test files are named with numbers.
171
172To run iotests, make sure QEMU is built successfully, then switch to the
173``tests/qemu-iotests`` directory under the build directory, and run ``./check``
174with desired arguments from there.
175
176By default, "raw" format and "file" protocol is used; all tests will be
177executed, except the unsupported ones. You can override the format and protocol
178with arguments:
179
180.. code::
181
182 # test with qcow2 format
183 ./check -qcow2
184 # or test a different protocol
185 ./check -nbd
186
187It's also possible to list test numbers explicitly:
188
189.. code::
190
191 # run selected cases with qcow2 format
192 ./check -qcow2 001 030 153
193
194Cache mode can be selected with the "-c" option, which may help reveal bugs
195that are specific to certain cache mode.
196
197More options are supported by the ``./check`` script, run ``./check -h`` for
198help.
199
200Writing a new test case
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200201~~~~~~~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800202
203Consider writing a tests case when you are making any changes to the block
204layer. An iotest case is usually the choice for that. There are already many
205test cases, so it is possible that extending one of them may achieve the goal
206and save the boilerplate to create one. (Unfortunately, there isn't a 100%
207reliable way to find a related one out of hundreds of tests. One approach is
208using ``git grep``.)
209
210Usually an iotest case consists of two files. One is an executable that
211produces output to stdout and stderr, the other is the expected reference
212output. They are given the same number in file names. E.g. Test script ``055``
213and reference output ``055.out``.
214
215In rare cases, when outputs differ between cache mode ``none`` and others, a
216``.out.nocache`` file is added. In other cases, when outputs differ between
217image formats, more than one ``.out`` files are created ending with the
218respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``.
219
220There isn't a hard rule about how to write a test script, but a new test is
221usually a (copy and) modification of an existing case. There are a few
222commonly used ways to create a test:
223
224* A Bash script. It will make use of several environmental variables related
225 to the testing procedure, and could source a group of ``common.*`` libraries
226 for some common helper routines.
227
228* A Python unittest script. Import ``iotests`` and create a subclass of
229 ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of
230 this approach is that the output is too scarce, and the script is considered
231 harder to debug.
232
233* A simple Python script without using unittest module. This could also import
234 ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit
235 from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest
236 execution. This is a combination of 1 and 2.
237
238Pick the language per your preference since both Bash and Python have
239comparable library support for invoking and interacting with QEMU programs. If
240you opt for Python, it is strongly recommended to write Python 3 compatible
241code.
242
Fam Zhengf4a1b652018-07-24 16:47:38 +0800243Both Python and Bash frameworks in iotests provide helpers to manage test
244images. They can be used to create and clean up images under the test
245directory. If no I/O or any protocol specific feature is needed, it is often
246more convenient to use the pseudo block driver, ``null-co://``, as the test
247image, which doesn't require image creation or cleaning up. Avoid system-wide
248devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``.
249Otherwise, image locking implications have to be considered. For example,
250another application on the host may have locked the file, possibly leading to a
251test failure. If using such devices are explicitly desired, consider adding
252``locking=off`` option to disable image locking.
253
Emanuele Giuseppe Esposito01937672021-08-09 11:01:02 +0200254Debugging a test case
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200255~~~~~~~~~~~~~~~~~~~~~
256
Emanuele Giuseppe Esposito01937672021-08-09 11:01:02 +0200257The following options to the ``check`` script can be useful when debugging
258a failing test:
259
Emanuele Giuseppe Espositoe92ecc32021-08-09 11:01:07 +0200260* ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a
261 connection from a gdb client. The options given to ``gdbserver`` (e.g. the
262 address on which to listen for connections) are taken from the ``$GDB_OPTIONS``
263 environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on
264 ``localhost:12345``.
265 It is possible to connect to it for example with
266 ``gdb -iex "target remote $addr"``, where ``$addr`` is the address
267 ``gdbserver`` listens on.
268 If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored,
269 regardless of whether it is set or not.
270
Emanuele Giuseppe Espositobd10a732021-08-09 11:01:12 +0200271* ``-valgrind`` attaches a valgrind instance to QEMU. If it detects
272 warnings, it will print and save the log in
273 ``$TEST_DIR/<valgrind_pid>.valgrind``.
274 The final command line will be ``valgrind --log-file=$TEST_DIR/
275 <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...``
276
Emanuele Giuseppe Esposito01937672021-08-09 11:01:02 +0200277* ``-d`` (debug) just increases the logging verbosity, showing
278 for example the QMP commands and answers.
279
Emanuele Giuseppe Esposito8ffcda22021-08-09 11:01:14 +0200280* ``-p`` (print) redirects QEMUs stdout and stderr to the test output,
281 instead of saving it into a log file in
282 ``$TEST_DIR/qemu-machine-<random_string>``.
283
Vladimir Sementsov-Ogievskiyb25a9482021-01-25 21:50:52 +0300284Test case groups
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200285~~~~~~~~~~~~~~~~
Vladimir Sementsov-Ogievskiyb25a9482021-01-25 21:50:52 +0300286
287"Tests may belong to one or more test groups, which are defined in the form
288of a comment in the test source file. By convention, test groups are listed
289in the second line of the test file, after the "#!/..." line, like this:
290
291.. code::
292
293 #!/usr/bin/env python3
294 # group: auto quick
295 #
296 ...
297
298Another way of defining groups is creating the tests/qemu-iotests/group.local
299file. This should be used only for downstream (this file should never appear
300in upstream). This file may be used for defining some downstream test groups
301or for temporarily disabling tests, like this:
302
303.. code::
304
305 # groups for some company downstream process
306 #
307 # ci - tests to run on build
308 # down - our downstream tests, not for upstream
309 #
310 # Format of each line is:
311 # TEST_NAME TEST_GROUP [TEST_GROUP ]...
312
313 013 ci
314 210 disabled
315 215 disabled
316 our-ugly-workaround-test down ci
317
318Note that the following group names have a special meaning:
319
320- quick: Tests in this group should finish within a few seconds.
321
322- auto: Tests in this group are used during "make check" and should be
323 runnable in any case. That means they should run with every QEMU binary
324 (also non-x86), with every QEMU configuration (i.e. must not fail if
325 an optional feature is not compiled in - but reporting a "skip" is ok),
326 work at least with the qcow2 file format, work with all kind of host
327 filesystems and users (e.g. "nobody" or "root") and must not take too
328 much memory and disk space (since CI pipelines tend to fail otherwise).
329
330- disabled: Tests in this group are disabled and ignored by check.
331
Alex Bennée663a0412021-02-22 10:14:53 +0000332.. _container-ref:
Alex Bennéef8ed3492019-09-19 14:36:35 +0100333
Alex Bennée663a0412021-02-22 10:14:53 +0000334Container based tests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200335---------------------
Fam Zheng4eb99562018-02-01 10:20:46 +0800336
337Introduction
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200338~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800339
Alex Bennée9c1f4912021-02-22 10:14:54 +0000340The container testing framework in QEMU utilizes public images to
341build and test QEMU in predefined and widely accessible Linux
342environments. This makes it possible to expand the test coverage
343across distros, toolchain flavors and library versions. The support
344was originally written for Docker although we also support Podman as
Matheus Tavares Bernardino0285a0e2022-08-23 10:46:19 -0300345an alternative container runtime. Although many of the target
Alex Bennée9c1f4912021-02-22 10:14:54 +0000346names and scripts are prefixed with "docker" the system will
347automatically run on whichever is configured.
Fam Zheng4eb99562018-02-01 10:20:46 +0800348
Alex Bennée4583cda2021-02-22 10:14:55 +0000349The container images are also used to augment the generation of tests
350for testing TCG. See :ref:`checktcg-ref` for more details.
351
Alex Bennée9c1f4912021-02-22 10:14:54 +0000352Docker Prerequisites
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200353~~~~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800354
355Install "docker" with the system package manager and start the Docker service
356on your development machine, then make sure you have the privilege to run
357Docker commands. Typically it means setting up passwordless ``sudo docker``
358command or login as root. For example:
359
360.. code::
361
362 $ sudo yum install docker
363 $ # or `apt-get install docker` for Ubuntu, etc.
364 $ sudo systemctl start docker
365 $ sudo docker ps
366
367The last command should print an empty table, to verify the system is ready.
368
369An alternative method to set up permissions is by adding the current user to
370"docker" group and making the docker daemon socket file (by default
371``/var/run/docker.sock``) accessible to the group:
372
373.. code::
374
375 $ sudo groupadd docker
Murilo Opsfelder Araujo29c33cc2019-02-07 16:43:46 -0200376 $ sudo usermod $USER -a -G docker
Fam Zheng4eb99562018-02-01 10:20:46 +0800377 $ sudo chown :docker /var/run/docker.sock
378
379Note that any one of above configurations makes it possible for the user to
380exploit the whole host with Docker bind mounting or other privileged
381operations. So only do it on development machines.
382
Alex Bennée9c1f4912021-02-22 10:14:54 +0000383Podman Prerequisites
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200384~~~~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800385
Alex Bennée9c1f4912021-02-22 10:14:54 +0000386Install "podman" with the system package manager.
Fam Zheng4eb99562018-02-01 10:20:46 +0800387
388.. code::
389
Alex Bennée9c1f4912021-02-22 10:14:54 +0000390 $ sudo dnf install podman
391 $ podman ps
Fam Zheng4eb99562018-02-01 10:20:46 +0800392
Alex Bennée9c1f4912021-02-22 10:14:54 +0000393The last command should print an empty table, to verify the system is ready.
394
395Quickstart
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200396~~~~~~~~~~
Alex Bennée9c1f4912021-02-22 10:14:54 +0000397
398From source tree, type ``make docker-help`` to see the help. Testing
399can be started without configuring or building QEMU (``configure`` and
400``make`` are done in the container, with parameters defined by the
401make target):
402
403.. code::
404
Alex Bennéecc1d2e02024-06-03 18:53:18 +0100405 make docker-test-build@debian
Alex Bennée9c1f4912021-02-22 10:14:54 +0000406
Alex Bennéecc1d2e02024-06-03 18:53:18 +0100407This will create a container instance using the ``debian`` image (the image
Fam Zheng4eb99562018-02-01 10:20:46 +0800408is downloaded and initialized automatically), in which the ``test-build`` job
409is executed.
410
Alex Bennée9c1f4912021-02-22 10:14:54 +0000411Registry
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200412~~~~~~~~
Alex Bennée9c1f4912021-02-22 10:14:54 +0000413
414The QEMU project has a container registry hosted by GitLab at
415``registry.gitlab.com/qemu-project/qemu`` which will automatically be
416used to pull in pre-built layers. This avoids unnecessary strain on
417the distro archives created by multiple developers running the same
418container build steps over and over again. This can be overridden
419locally by using the ``NOCACHE`` build option:
420
421.. code::
422
Alex Bennéed996f0a2022-09-14 16:59:50 +0100423 make docker-image-debian-arm64-cross NOCACHE=1
Alex Bennée9c1f4912021-02-22 10:14:54 +0000424
Fam Zheng4eb99562018-02-01 10:20:46 +0800425Images
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200426~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800427
Alex Bennéecc1d2e02024-06-03 18:53:18 +0100428Along with many other images, the ``debian`` image is defined in a Dockerfile
429in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help``
Fam Zheng4eb99562018-02-01 10:20:46 +0800430command will list all the available images.
431
Fam Zheng4eb99562018-02-01 10:20:46 +0800432A ``.pre`` script can be added beside the ``.docker`` file, which will be
433executed before building the image under the build context directory. This is
434mainly used to do necessary host side setup. One such setup is ``binfmt_misc``,
435for example, to make qemu-user powered cross build containers work.
436
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000437Most of the existing Dockerfiles were written by hand, simply by creating a
438a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory.
439This has led to an inconsistent set of packages being present across the
440different containers.
441
442Thus going forward, QEMU is aiming to automatically generate the Dockerfiles
443using the ``lcitool`` program provided by the ``libvirt-ci`` project:
444
445 https://gitlab.com/libvirt/libvirt-ci
446
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100447``libvirt-ci`` contains an ``lcitool`` program as well as a list of
448mappings to distribution package names for a wide variety of third
449party projects. ``lcitool`` applies the mappings to a list of build
450pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the
451list of native packages to install on each distribution, and uses them
452to generate build environments (dockerfiles and Cirrus CI variable files)
453that are consistent across OS distribution.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000454
455
456Adding new build pre-requisites
457^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
458
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100459When preparing a patch series that adds a new build
460pre-requisite to QEMU, the prerequisites should to be added to
461``tests/lcitool/projects/qemu.yml`` in order to make the dependency
462available in the CI build environments.
463
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000464In the simple case where the pre-requisite is already known to ``libvirt-ci``
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100465the following steps are needed:
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000466
467 * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite
468
469 * Run ``make lcitool-refresh`` to re-generate all relevant build environment
470 manifests
471
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100472It may be that ``libvirt-ci`` does not know about the new pre-requisite.
473If that is the case, some extra preparation steps will be required
474first to contribute the mapping to the ``libvirt-ci`` project:
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000475
476 * Fork the ``libvirt-ci`` project on gitlab
477
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100478 * Add an entry for the new build prerequisite to
479 ``lcitool/facts/mappings.yml``, listing its native package name on as
480 many OS distros as practical. Run ``python -m pytest --regenerate-output``
481 and check that the changes are correct.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000482
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100483 * Commit the ``mappings.yml`` change together with the regenerated test
484 files, and submit a merge request to the ``libvirt-ci`` project.
485 Please note in the description that this is a new build pre-requisite
486 desired for use with QEMU.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000487
488 * CI pipeline will run to validate that the changes to ``mappings.yml``
489 are correct, by attempting to install the newly listed package on
490 all OS distributions supported by ``libvirt-ci``.
491
492 * Once the merge request is accepted, go back to QEMU and update
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100493 the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that
494 contains the ``mappings.yml`` update. Then add the prerequisite and
495 run ``make lcitool-refresh``.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000496
Ani Sinha2a851fc2023-05-06 12:50:12 +0530497 * Please also trigger gitlab container generation pipelines on your change
498 for as many OS distros as practical to make sure that there are no
499 obvious breakages when adding the new pre-requisite. Please see
500 `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation
501 page on how to trigger gitlab CI pipelines on your change.
502
Paolo Bonzini32c06132022-10-12 18:20:44 +0200503For enterprise distros that default to old, end-of-life versions of the
504Python runtime, QEMU uses a separate set of mappings that work with more
505recent versions. These can be found in ``tests/lcitool/mappings.yml``.
506Modifying this file should not be necessary unless the new pre-requisite
507is a Python library or tool.
508
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000509
510Adding new OS distros
511^^^^^^^^^^^^^^^^^^^^^
512
513In some cases ``libvirt-ci`` will not know about the OS distro that is
514desired to be tested. Before adding a new OS distro, discuss the proposed
515addition:
516
517 * Send a mail to qemu-devel, copying people listed in the
518 MAINTAINERS file for ``Build and test automation``.
519
520 There are limited CI compute resources available to QEMU, so the
521 cost/benefit tradeoff of adding new OS distros needs to be considered.
522
523 * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues
524 pointing to the qemu-devel mail thread in the archives.
525
526 This alerts other people who might be interested in the work
527 to avoid duplication, as well as to get feedback from libvirt-ci
528 maintainers on any tips to ease the addition
529
530Assuming there is agreement to add a new OS distro then
531
532 * Fork the ``libvirt-ci`` project on gitlab
533
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100534 * Add metadata under ``lcitool/facts/targets/`` for the new OS
535 distro. There might be code changes required if the OS distro
536 uses a package format not currently known. The ``libvirt-ci``
537 maintainers can advise on this when the issue is filed.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000538
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100539 * Edit the ``lcitool/facts/mappings.yml`` change to add entries for
540 the new OS, listing the native package names for as many packages
541 as practical. Run ``python -m pytest --regenerate-output`` and
542 check that the changes are correct.
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000543
Paolo Bonzinifa1ce1d2023-02-09 12:24:26 +0100544 * Commit the changes to ``lcitool/facts`` and the regenerated test
545 files, and submit a merge request to the ``libvirt-ci`` project.
546 Please note in the description that this is a new build pre-requisite
547 desired for use with QEMU
Daniel P. Berrangé4ebb0402022-01-05 13:49:42 +0000548
549 * CI pipeline will run to validate that the changes to ``mappings.yml``
550 are correct, by attempting to install the newly listed package on
551 all OS distributions supported by ``libvirt-ci``.
552
553 * Once the merge request is accepted, go back to QEMU and update
554 the ``libvirt-ci`` submodule to point to a commit that contains
555 the ``mappings.yml`` update.
556
557
Fam Zheng4eb99562018-02-01 10:20:46 +0800558Tests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200559~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800560
561Different tests are added to cover various configurations to build and test
562QEMU. Docker tests are the executables under ``tests/docker`` named
563``test-*``. They are typically shell scripts and are built on top of a shell
564library, ``tests/docker/common.rc``, which provides helpers to find the QEMU
565source and build it.
566
Alex Bennée9c1f4912021-02-22 10:14:54 +0000567The full list of tests is printed in the ``make docker-help`` help.
Fam Zheng4eb99562018-02-01 10:20:46 +0800568
Fam Zheng4eb99562018-02-01 10:20:46 +0800569Debugging a Docker test failure
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200570~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800571
572When CI tasks, maintainers or yourself report a Docker test failure, follow the
573below steps to debug it:
574
5751. Locally reproduce the failure with the reported command line. E.g. run
Philippe Mathieu-Daudé098bfc02023-06-24 14:58:24 +0200576 ``make docker-test-mingw@fedora-win64-cross J=8``.
Fam Zheng4eb99562018-02-01 10:20:46 +08005772. Add "V=1" to the command line, try again, to see the verbose output.
5783. Further add "DEBUG=1" to the command line. This will pause in a shell prompt
579 in the container right before testing starts. You could either manually
580 build QEMU and run tests from there, or press Ctrl-D to let the Docker
581 testing continue.
5824. If you press Ctrl-D, the same building and testing procedure will begin, and
583 will hopefully run into the error again. After that, you will be dropped to
584 the prompt for debug.
585
586Options
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200587~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800588
589Various options can be used to affect how Docker tests are done. The full
590list is in the ``make docker`` help text. The frequently used ones are:
591
592* ``V=1``: the same as in top level ``make``. It will be propagated to the
593 container and enable verbose output.
594* ``J=$N``: the number of parallel tasks in make commands in the container,
595 similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in
596 top level ``make`` will not be propagated into the container.)
597* ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test
598 failure" section.
599
Robert Foley3b6882b2020-06-12 20:02:34 +0100600Thread Sanitizer
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200601----------------
Robert Foley3b6882b2020-06-12 20:02:34 +0100602
603Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports
604building and testing with this tool.
605
606For more information on TSan:
607
608https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual
609
610Thread Sanitizer in Docker
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200611~~~~~~~~~~~~~~~~~~~~~~~~~~
Alex Bennée171080d2023-02-28 19:06:37 +0000612TSan is currently supported in the ubuntu2204 docker.
Robert Foley3b6882b2020-06-12 20:02:34 +0100613
614The test-tsan test will build using TSan and then run make check.
615
616.. code::
617
Alex Bennée171080d2023-02-28 19:06:37 +0000618 make docker-test-tsan@ubuntu2204
Robert Foley3b6882b2020-06-12 20:02:34 +0100619
620TSan warnings under docker are placed in files located at build/tsan/.
621
622We recommend using DEBUG=1 to allow launching the test from inside the docker,
623and to allow review of the warnings generated by TSan.
624
625Building and Testing with TSan
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200626~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Robert Foley3b6882b2020-06-12 20:02:34 +0100627
628It is possible to build and test with TSan, with a few additional steps.
629These steps are normally done automatically in the docker.
630
631There is a one time patch needed in clang-9 or clang-10 at this time:
632
633.. code::
634
635 sed -i 's/^const/static const/g' \
636 /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h
637
638To configure the build for TSan:
639
640.. code::
641
642 ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \
643 --disable-werror --extra-cflags="-O0"
644
645The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment
646variable.
647
648More information on the TSAN_OPTIONS can be found here:
649
650https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
651
652For example:
653
654.. code::
655
656 export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \
657 detect_deadlocks=false history_size=7 exitcode=0 \
658 log_path=<build path>/tsan/tsan_warning
659
660The above exitcode=0 has TSan continue without error if any warnings are found.
661This allows for running the test and then checking the warnings afterwards.
662If you want TSan to stop and exit with error on warnings, use exitcode=66.
663
664TSan Suppressions
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200665~~~~~~~~~~~~~~~~~
Robert Foley3b6882b2020-06-12 20:02:34 +0100666Keep in mind that for any data race warning, although there might be a data race
667detected by TSan, there might be no actual bug here. TSan provides several
668different mechanisms for suppressing warnings. In general it is recommended
669to fix the code if possible to eliminate the data race rather than suppress
670the warning.
671
672A few important files for suppressing warnings are:
673
674tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime.
zhaolichang76ca4b52020-09-17 15:50:22 +0800675The comment on each suppression will typically indicate why we are
Robert Foley3b6882b2020-06-12 20:02:34 +0100676suppressing it. More information on the file format can be found here:
677
678https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions
679
Thomas Huthf9a19bd2023-11-09 18:47:20 +0100680tests/tsan/ignore.tsan - Has TSan warnings we wish to disable
Robert Foley3b6882b2020-06-12 20:02:34 +0100681at compile time for test or debug.
682Add flags to configure to enable:
683
Thomas Huthf9a19bd2023-11-09 18:47:20 +0100684"--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan"
Robert Foley3b6882b2020-06-12 20:02:34 +0100685
686More information on the file format can be found here under "Blacklist Format":
687
688https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags
689
690TSan Annotations
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200691~~~~~~~~~~~~~~~~
Robert Foley3b6882b2020-06-12 20:02:34 +0100692include/qemu/tsan.h defines annotations. See this file for more descriptions
693of the annotations themselves. Annotations can be used to suppress
694TSan warnings or give TSan more information so that it can detect proper
695relationships between accesses of data.
696
697Annotation examples can be found here:
698
699https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/
700
701Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp
702
703The full set of annotations can be found here:
704
705https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp
706
Alex Bennée396408e2022-04-19 10:09:57 +0100707docker-binfmt-image-debian-% targets
708------------------------------------
709
710It is possible to combine Debian's bootstrap scripts with a configured
711``binfmt_misc`` to bootstrap a number of Debian's distros including
712experimental ports not yet supported by a released OS. This can
713simplify setting up a rootfs by using docker to contain the foreign
714rootfs rather than manually invoking chroot.
715
716Setting up ``binfmt_misc``
717~~~~~~~~~~~~~~~~~~~~~~~~~~
718
719You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU
720user binary to automatically run binaries for the foreign
721architecture. While the scripts will try their best to work with
722dynamically linked QEMU's a statically linked one will present less
723potential complications when copying into the docker image. Modern
724kernels support the ``F`` (fix binary) flag which will open the QEMU
725executable on setup and avoids the need to find and re-open in the
726chroot environment. This is triggered with the ``--persistent`` flag.
727
728Example invocation
729~~~~~~~~~~~~~~~~~~
730
731For example to setup the HPPA ports builds of Debian::
732
733 make docker-binfmt-image-debian-sid-hppa \
734 DEB_TYPE=sid DEB_ARCH=hppa \
735 DEB_URL=http://ftp.ports.debian.org/debian-ports/ \
736 DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \
737 EXECUTABLE=(pwd)/qemu-hppa V=1
738
739The ``DEB_`` variables are substitutions used by
Manos Pitsidianakisc555b502024-02-20 10:52:08 +0200740``debian-bootstrap.pre`` which is called to do the initial debootstrap
Alex Bennée396408e2022-04-19 10:09:57 +0100741of the rootfs before it is copied into the container. The second stage
742is run as part of the build. The final image will be tagged as
743``qemu/debian-sid-hppa``.
744
Fam Zheng4eb99562018-02-01 10:20:46 +0800745VM testing
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200746----------
Fam Zheng4eb99562018-02-01 10:20:46 +0800747
748This test suite contains scripts that bootstrap various guest images that have
749necessary packages to build QEMU. The basic usage is documented in ``Makefile``
Philippe Mathieu-Daudé4f2f6272019-05-31 08:43:41 +0200750help which is displayed with ``make vm-help``.
Fam Zheng4eb99562018-02-01 10:20:46 +0800751
752Quickstart
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200753~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800754
Philippe Mathieu-Daudé4f2f6272019-05-31 08:43:41 +0200755Run ``make vm-help`` to list available make targets. Invoke a specific make
Fam Zheng4eb99562018-02-01 10:20:46 +0800756command to run build test in an image. For example, ``make vm-build-freebsd``
757will build the source tree in the FreeBSD image. The command can be executed
758from either the source tree or the build dir; if the former, ``./configure`` is
759not needed. The command will then generate the test image in ``./tests/vm/``
760under the working directory.
761
762Note: images created by the scripts accept a well-known RSA key pair for SSH
763access, so they SHOULD NOT be exposed to external interfaces if you are
764concerned about attackers taking control of the guest and potentially
765exploiting a QEMU security bug to compromise the host.
766
Wainer dos Santos Moschetta1e489312019-11-14 08:42:46 -0500767QEMU binaries
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200768~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800769
Philippe Mathieu-Daudéc5ba6212021-11-18 20:27:44 +0100770By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If
771there isn't one, or if it is older than 2.10, the test won't work. In this case,
Fam Zheng4eb99562018-02-01 10:20:46 +0800772provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``.
773
Philippe Mathieu-Daudéc5ba6212021-11-18 20:27:44 +0100774Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable.
Wainer dos Santos Moschetta1e489312019-11-14 08:42:46 -0500775
Fam Zheng4eb99562018-02-01 10:20:46 +0800776Make jobs
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200777~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800778
779The ``-j$X`` option in the make command line is not propagated into the VM,
780specify ``J=$X`` to control the make jobs in the guest.
781
782Debugging
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200783~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800784
785Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive
786debugging and verbose output. If this is not enough, see the next section.
Peter Maydell41e33402018-08-03 09:52:28 +0100787``V=1`` will be propagated down into the make jobs in the guest.
Fam Zheng4eb99562018-02-01 10:20:46 +0800788
789Manual invocation
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200790~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800791
792Each guest script is an executable script with the same command line options.
793For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``:
794
795.. code::
796
797 $ cd $QEMU_SRC/tests/vm
798
799 # To bootstrap the image
800 $ ./netbsd --build-image --image /var/tmp/netbsd.img
801 <...>
802
803 # To run an arbitrary command in guest (the output will not be echoed unless
804 # --debug is added)
805 $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a
806
807 # To build QEMU in guest
808 $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC
809
810 # To get to an interactive shell
811 $ ./netbsd --interactive --image /var/tmp/netbsd.img sh
812
813Adding new guests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200814~~~~~~~~~~~~~~~~~
Fam Zheng4eb99562018-02-01 10:20:46 +0800815
816Please look at existing guest scripts for how to add new guests.
817
818Most importantly, create a subclass of BaseVM and implement ``build_image()``
819method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from
820the script's ``main()``.
821
822* Usually in ``build_image()``, a template image is downloaded from a
823 predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and
824 the checksum, so consider using it.
825
826* Once the image is downloaded, users, SSH server and QEMU build deps should
827 be set up:
828
829 - Root password set to ``BaseVM.ROOT_PASS``
830 - User ``BaseVM.GUEST_USER`` is created, and password set to
831 ``BaseVM.GUEST_PASS``
832 - SSH service is enabled and started on boot,
833 ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys``
834 file of both root and the normal user
835 - DHCP client service is enabled and started on boot, so that it can
836 automatically configure the virtio-net-pci NIC and communicate with QEMU
837 user net (10.0.2.2)
838 - Necessary packages are installed to untar the source tarball and build
839 QEMU
840
841* Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that
842 untars a raw virtio-blk block device, which is the tarball data blob of the
843 QEMU source tree, then configure/build it. Running "make check" is also
844 recommended.
845
846Image fuzzer testing
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200847--------------------
Fam Zheng4eb99562018-02-01 10:20:46 +0800848
849An image fuzzer was added to exercise format drivers. Currently only qcow2 is
850supported. To start the fuzzer, run
851
852.. code::
853
854 tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2
855
Philippe Mathieu-Daudéc5ba6212021-11-18 20:27:44 +0100856Alternatively, some command different from ``qemu-img info`` can be tested, by
Fam Zheng4eb99562018-02-01 10:20:46 +0800857changing the ``-c`` option.
Cleber Rosac3d7e8c2018-05-30 14:41:52 -0400858
Thomas Huthc3e24cf2024-08-30 15:38:38 +0200859Functional tests using Python
860-----------------------------
861
862The ``tests/functional`` directory hosts functional tests written in
863Python. You can run the functional tests simply by executing:
864
865.. code::
866
867 make check-functional
868
869See :ref:`checkfunctional-ref` for more details.
870
Willian Rampazzobbbd9b62021-11-05 12:53:54 -0300871Integration tests using the Avocado Framework
872---------------------------------------------
Cleber Rosac3d7e8c2018-05-30 14:41:52 -0400873
Willian Rampazzobbbd9b62021-11-05 12:53:54 -0300874The ``tests/avocado`` directory hosts integration tests. They're usually
875higher level tests, and may interact with external resources and with
876various guest operating systems.
Cleber Rosac3d7e8c2018-05-30 14:41:52 -0400877
Willian Rampazzobbbd9b62021-11-05 12:53:54 -0300878You can run the avocado tests simply by executing:
Cleber Rosaa56931e2018-10-18 11:31:33 -0400879
880.. code::
881
Willian Rampazzobbbd9b62021-11-05 12:53:54 -0300882 make check-avocado
Cleber Rosaa56931e2018-10-18 11:31:33 -0400883
Thomas Huth2133c2a2024-08-30 15:38:36 +0200884See :ref:`checkavocado-ref` for more details.
Cleber Rosaa56931e2018-10-18 11:31:33 -0400885
Alex Bennéef8ed3492019-09-19 14:36:35 +0100886
Alex Bennée4583cda2021-02-22 10:14:55 +0000887.. _checktcg-ref:
888
Alex Bennéef8ed3492019-09-19 14:36:35 +0100889Testing with "make check-tcg"
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200890-----------------------------
Alex Bennéef8ed3492019-09-19 14:36:35 +0100891
892The check-tcg tests are intended for simple smoke tests of both
893linux-user and softmmu TCG functionality. However to build test
894programs for guest targets you need to have cross compilers available.
895If your distribution supports cross compilers you can do something as
896simple as::
897
898 apt install gcc-aarch64-linux-gnu
899
900The configure script will automatically pick up their presence.
901Sometimes compilers have slightly odd names so the availability of
902them can be prompted by passing in the appropriate configure option
903for the architecture in question, for example::
904
905 $(configure) --cross-cc-aarch64=aarch64-cc
906
Matheus Ferst479ca4c2022-01-20 14:31:41 -0300907There is also a ``--cross-cc-cflags-ARCH`` flag in case additional
Alex Bennéef8ed3492019-09-19 14:36:35 +0100908compiler flags are needed to build for a given target.
909
Alex Bennée663a0412021-02-22 10:14:53 +0000910If you have the ability to run containers as the user the build system
911will automatically use them where no system compiler is available. For
912architectures where we also support building QEMU we will generally
913use the same container to build tests. However there are a number of
914additional containers defined that have a minimal cross-build
915environment that is only suitable for building test cases. Sometimes
916we may use a bleeding edge distribution for compiler features needed
917for test cases that aren't yet in the LTS distros we support for QEMU
918itself.
919
920See :ref:`container-ref` for more details.
Alex Bennéef8ed3492019-09-19 14:36:35 +0100921
922Running subset of tests
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200923~~~~~~~~~~~~~~~~~~~~~~~
Alex Bennéef8ed3492019-09-19 14:36:35 +0100924
925You can build the tests for one architecture::
926
927 make build-tcg-tests-$TARGET
928
929And run with::
930
931 make run-tcg-tests-$TARGET
932
933Adding ``V=1`` to the invocation will show the details of how to
934invoke QEMU for the test which is useful for debugging tests.
935
Alex Bennéeb4212062024-07-29 15:44:07 +0100936Running individual tests
937~~~~~~~~~~~~~~~~~~~~~~~~
938
939Tests can also be run directly from the test build directory. If you
940run ``make help`` from the test build directory you will get a list of
941all the tests that can be run. Please note that same binaries are used
942in multiple tests, for example::
943
944 make run-plugin-test-mmap-with-libinline.so
945
946will run the mmap test with the ``libinline.so`` TCG plugin. The
947gdbstub tests also re-use the test binaries but while exercising gdb.
948
Alex Bennéef8ed3492019-09-19 14:36:35 +0100949TCG test dependencies
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200950~~~~~~~~~~~~~~~~~~~~~
Alex Bennéef8ed3492019-09-19 14:36:35 +0100951
952The TCG tests are deliberately very light on dependencies and are
Philippe Mathieu-Daudé7893e422023-10-04 11:06:20 +0200953either totally bare with minimal gcc lib support (for system-mode tests)
Alex Bennéef8ed3492019-09-19 14:36:35 +0100954or just glibc (for linux-user tests). This is because getting a cross
955compiler to work with additional libraries can be challenging.
956
957Other TCG Tests
958---------------
959
960There are a number of out-of-tree test suites that are used for more
961extensive testing of processor features.
962
963KVM Unit Tests
964~~~~~~~~~~~~~~
965
966The KVM unit tests are designed to run as a Guest OS under KVM but
967there is no reason why they can't exercise the TCG as well. It
968provides a minimal OS kernel with hooks for enabling the MMU as well
969as reporting test results via a special device::
970
971 https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git
972
973Linux Test Project
974~~~~~~~~~~~~~~~~~~
975
976The LTP is focused on exercising the syscall interface of a Linux
977kernel. It checks that syscalls behave as documented and strives to
978exercise as many corner cases as possible. It is a useful test suite
979to run to exercise QEMU's linux-user code::
980
981 https://linux-test-project.github.io/
Paolo Bonzini9fce3602021-09-22 17:06:57 +0200982
983GCC gcov support
Paolo Bonzini16e79e12021-09-07 17:21:58 +0200984----------------
Paolo Bonzini9fce3602021-09-22 17:06:57 +0200985
986``gcov`` is a GCC tool to analyze the testing coverage by
987instrumenting the tested code. To use it, configure QEMU with
988``--enable-gcov`` option and build. Then run the tests as usual.
989
990If you want to gather coverage information on a single test the ``make
991clean-gcda`` target can be used to delete any existing coverage
992information before running a single test.
993
994You can generate a HTML coverage report by executing ``make
995coverage-html`` which will create
996``meson-logs/coveragereport/index.html``.
997
998Further analysis can be conducted by running the ``gcov`` command
999directly on the various .gcda output files. Please read the ``gcov``
1000documentation for more information.