Alex Bennée | 7266ecc | 2022-05-27 16:36:03 +0100 | [diff] [blame] | 1 | .. _testing: |
| 2 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 3 | Testing in QEMU |
| 4 | =============== |
| 5 | |
Alex Bennée | c8d6e04 | 2024-07-29 15:44:06 +0100 | [diff] [blame] | 6 | QEMU's testing infrastructure is fairly complex as it covers |
| 7 | everything from unit testing and exercising specific sub-systems all |
| 8 | the way to full blown acceptance tests. To get an overview of the |
| 9 | tests you can run ``make check-help`` from either the source or build |
| 10 | tree. |
| 11 | |
| 12 | Most (but not all) tests are also integrated into the meson build |
| 13 | system so can be run directly from the build tree, for example: |
| 14 | |
| 15 | .. code:: |
| 16 | |
| 17 | [./pyvenv/bin/]meson test --suite qemu:softfloat |
| 18 | |
| 19 | will run just the softfloat tests. |
| 20 | |
| 21 | The rest of this document will cover the details for specific test |
| 22 | groups. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 23 | |
| 24 | Testing with "make check" |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 25 | ------------------------- |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 26 | |
Alex Bennée | c8d6e04 | 2024-07-29 15:44:06 +0100 | [diff] [blame] | 27 | The "make check" testing family includes most of the C based tests in QEMU. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 28 | |
| 29 | The usual way to run these tests is: |
| 30 | |
| 31 | .. code:: |
| 32 | |
| 33 | make check |
| 34 | |
Thomas Huth | 316082b | 2020-01-22 14:40:20 +0100 | [diff] [blame] | 35 | which includes QAPI schema tests, unit tests, QTests and some iotests. |
| 36 | Different sub-types of "make check" tests will be explained below. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 37 | |
| 38 | Before running tests, it is best to build QEMU programs first. Some tests |
| 39 | expect the executables to exist and will fail with obscure messages if they |
| 40 | cannot find them. |
| 41 | |
| 42 | Unit tests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 43 | ~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 44 | |
| 45 | Unit tests, which can be invoked with ``make check-unit``, are simple C tests |
| 46 | that typically link to individual QEMU object files and exercise them by |
| 47 | calling exported functions. |
| 48 | |
| 49 | If you are writing new code in QEMU, consider adding a unit test, especially |
| 50 | for utility modules that are relatively stateless or have few dependencies. To |
| 51 | add a new unit test: |
| 52 | |
Wainer dos Santos Moschetta | 8db5c3e | 2021-03-18 14:44:07 -0300 | [diff] [blame] | 53 | 1. Create a new source file. For example, ``tests/unit/foo-test.c``. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 54 | |
| 55 | 2. Write the test. Normally you would include the header file which exports |
| 56 | the module API, then verify the interface behaves as expected from your |
| 57 | test. The test code should be organized with the glib testing framework. |
| 58 | Copying and modifying an existing test is usually a good idea. |
| 59 | |
Wainer dos Santos Moschetta | 8db5c3e | 2021-03-18 14:44:07 -0300 | [diff] [blame] | 60 | 3. Add the test to ``tests/unit/meson.build``. The unit tests are listed in a |
Paolo Bonzini | bab88ea | 2020-10-06 08:49:55 -0400 | [diff] [blame] | 61 | dictionary called ``tests``. The values are any additional sources and |
| 62 | dependencies to be linked with the test. For a simple test whose source |
Wainer dos Santos Moschetta | 8db5c3e | 2021-03-18 14:44:07 -0300 | [diff] [blame] | 63 | is in ``tests/unit/foo-test.c``, it is enough to add an entry like:: |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 64 | |
Paolo Bonzini | bab88ea | 2020-10-06 08:49:55 -0400 | [diff] [blame] | 65 | { |
| 66 | ... |
| 67 | 'foo-test': [], |
| 68 | ... |
| 69 | } |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 70 | |
| 71 | Since unit tests don't require environment variables, the simplest way to debug |
| 72 | a unit test failure is often directly invoking it or even running it under |
| 73 | ``gdb``. However there can still be differences in behavior between ``make`` |
| 74 | invocations and your manual run, due to ``$MALLOC_PERTURB_`` environment |
| 75 | variable (which affects memory reclamation and catches invalid pointers better) |
| 76 | and gtester options. If necessary, you can run |
| 77 | |
| 78 | .. code:: |
Cleber Rosa | 9297081 | 2018-10-04 12:18:47 -0400 | [diff] [blame] | 79 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 80 | make check-unit V=1 |
| 81 | |
| 82 | and copy the actual command line which executes the unit test, then run |
| 83 | it from the command line. |
| 84 | |
| 85 | QTest |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 86 | ~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 87 | |
| 88 | QTest is a device emulation testing framework. It can be very useful to test |
| 89 | device models; it could also control certain aspects of QEMU (such as virtual |
Eduardo Habkost | a738a50 | 2020-10-05 16:52:26 -0400 | [diff] [blame] | 90 | clock stepping), with a special purpose "qtest" protocol. Refer to |
| 91 | :doc:`qtest` for more details. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 92 | |
| 93 | QTest cases can be executed with |
| 94 | |
| 95 | .. code:: |
| 96 | |
| 97 | make check-qtest |
| 98 | |
Bin Meng | 0b49bc1 | 2022-09-27 19:06:32 +0800 | [diff] [blame] | 99 | Writing portable test cases |
| 100 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 101 | Both unit tests and qtests can run on POSIX hosts as well as Windows hosts. |
| 102 | Care must be taken when writing portable test cases that can be built and run |
| 103 | successfully on various hosts. The following list shows some best practices: |
| 104 | |
| 105 | * Use portable APIs from glib whenever necessary, e.g.: g_setenv(), |
| 106 | g_mkdtemp(), g_mkdir(). |
| 107 | * Avoid using hardcoded /tmp for temporary file directory. |
| 108 | Use g_get_tmp_dir() instead. |
| 109 | * Bear in mind that Windows has different special string representation for |
| 110 | stdin/stdout/stderr and null devices. For example if your test case uses |
| 111 | "/dev/fd/2" and "/dev/null" on Linux, remember to use "2" and "nul" on |
| 112 | Windows instead. Also IO redirection does not work on Windows, so avoid |
| 113 | using "2>nul" whenever necessary. |
| 114 | * If your test cases uses the blkdebug feature, use relative path to pass |
| 115 | the config and image file paths in the command line as Windows absolute |
| 116 | path contains the delimiter ":" which will confuse the blkdebug parser. |
Stefan Weil | 1e458f1 | 2022-10-30 11:59:44 +0100 | [diff] [blame] | 117 | * Use double quotes in your extra QEMU command line in your test cases |
Bin Meng | 0b49bc1 | 2022-09-27 19:06:32 +0800 | [diff] [blame] | 118 | instead of single quotes, as Windows does not drop single quotes when |
| 119 | passing the command line to QEMU. |
| 120 | * Windows opens a file in text mode by default, while a POSIX compliant |
| 121 | implementation treats text files and binary files the same. So if your |
| 122 | test cases opens a file to write some data and later wants to compare the |
| 123 | written data with the original one, be sure to pass the letter 'b' as |
| 124 | part of the mode string to fopen(), or O_BINARY flag for the open() call. |
| 125 | * If a certain test case can only run on POSIX or Linux hosts, use a proper |
| 126 | #ifdef in the codes. If the whole test suite cannot run on Windows, disable |
| 127 | the build in the meson.build file. |
| 128 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 129 | QAPI schema tests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 130 | ~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 131 | |
| 132 | The QAPI schema tests validate the QAPI parser used by QMP, by feeding |
| 133 | predefined input to the parser and comparing the result with the reference |
| 134 | output. |
| 135 | |
| 136 | The input/output data is managed under the ``tests/qapi-schema`` directory. |
| 137 | Each test case includes four files that have a common base name: |
| 138 | |
| 139 | * ``${casename}.json`` - the file contains the JSON input for feeding the |
| 140 | parser |
| 141 | * ``${casename}.out`` - the file contains the expected stdout from the parser |
| 142 | * ``${casename}.err`` - the file contains the expected stderr from the parser |
| 143 | * ``${casename}.exit`` - the expected error code |
| 144 | |
| 145 | Consider adding a new QAPI schema test when you are making a change on the QAPI |
| 146 | parser (either fixing a bug or extending/modifying the syntax). To do this: |
| 147 | |
| 148 | 1. Add four files for the new case as explained above. For example: |
| 149 | |
| 150 | ``$EDITOR tests/qapi-schema/foo.{json,out,err,exit}``. |
| 151 | |
| 152 | 2. Add the new test in ``tests/Makefile.include``. For example: |
| 153 | |
| 154 | ``qapi-schema += foo.json`` |
| 155 | |
| 156 | check-block |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 157 | ~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 158 | |
Thomas Huth | 316082b | 2020-01-22 14:40:20 +0100 | [diff] [blame] | 159 | ``make check-block`` runs a subset of the block layer iotests (the tests that |
Vladimir Sementsov-Ogievskiy | b25a948 | 2021-01-25 21:50:52 +0300 | [diff] [blame] | 160 | are in the "auto" group). |
Thomas Huth | 316082b | 2020-01-22 14:40:20 +0100 | [diff] [blame] | 161 | See the "QEMU iotests" section below for more information. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 162 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 163 | QEMU iotests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 164 | ------------ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 165 | |
| 166 | QEMU iotests, under the directory ``tests/qemu-iotests``, is the testing |
| 167 | framework widely used to test block layer related features. It is higher level |
| 168 | than "make check" tests and 99% of the code is written in bash or Python |
| 169 | scripts. The testing success criteria is golden output comparison, and the |
| 170 | test files are named with numbers. |
| 171 | |
| 172 | To run iotests, make sure QEMU is built successfully, then switch to the |
| 173 | ``tests/qemu-iotests`` directory under the build directory, and run ``./check`` |
| 174 | with desired arguments from there. |
| 175 | |
| 176 | By default, "raw" format and "file" protocol is used; all tests will be |
| 177 | executed, except the unsupported ones. You can override the format and protocol |
| 178 | with arguments: |
| 179 | |
| 180 | .. code:: |
| 181 | |
| 182 | # test with qcow2 format |
| 183 | ./check -qcow2 |
| 184 | # or test a different protocol |
| 185 | ./check -nbd |
| 186 | |
| 187 | It's also possible to list test numbers explicitly: |
| 188 | |
| 189 | .. code:: |
| 190 | |
| 191 | # run selected cases with qcow2 format |
| 192 | ./check -qcow2 001 030 153 |
| 193 | |
| 194 | Cache mode can be selected with the "-c" option, which may help reveal bugs |
| 195 | that are specific to certain cache mode. |
| 196 | |
| 197 | More options are supported by the ``./check`` script, run ``./check -h`` for |
| 198 | help. |
| 199 | |
| 200 | Writing a new test case |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 201 | ~~~~~~~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 202 | |
| 203 | Consider writing a tests case when you are making any changes to the block |
| 204 | layer. An iotest case is usually the choice for that. There are already many |
| 205 | test cases, so it is possible that extending one of them may achieve the goal |
| 206 | and save the boilerplate to create one. (Unfortunately, there isn't a 100% |
| 207 | reliable way to find a related one out of hundreds of tests. One approach is |
| 208 | using ``git grep``.) |
| 209 | |
| 210 | Usually an iotest case consists of two files. One is an executable that |
| 211 | produces output to stdout and stderr, the other is the expected reference |
| 212 | output. They are given the same number in file names. E.g. Test script ``055`` |
| 213 | and reference output ``055.out``. |
| 214 | |
| 215 | In rare cases, when outputs differ between cache mode ``none`` and others, a |
| 216 | ``.out.nocache`` file is added. In other cases, when outputs differ between |
| 217 | image formats, more than one ``.out`` files are created ending with the |
| 218 | respective format names, e.g. ``178.out.qcow2`` and ``178.out.raw``. |
| 219 | |
| 220 | There isn't a hard rule about how to write a test script, but a new test is |
| 221 | usually a (copy and) modification of an existing case. There are a few |
| 222 | commonly used ways to create a test: |
| 223 | |
| 224 | * A Bash script. It will make use of several environmental variables related |
| 225 | to the testing procedure, and could source a group of ``common.*`` libraries |
| 226 | for some common helper routines. |
| 227 | |
| 228 | * A Python unittest script. Import ``iotests`` and create a subclass of |
| 229 | ``iotests.QMPTestCase``, then call ``iotests.main`` method. The downside of |
| 230 | this approach is that the output is too scarce, and the script is considered |
| 231 | harder to debug. |
| 232 | |
| 233 | * A simple Python script without using unittest module. This could also import |
| 234 | ``iotests`` for launching QEMU and utilities etc, but it doesn't inherit |
| 235 | from ``iotests.QMPTestCase`` therefore doesn't use the Python unittest |
| 236 | execution. This is a combination of 1 and 2. |
| 237 | |
| 238 | Pick the language per your preference since both Bash and Python have |
| 239 | comparable library support for invoking and interacting with QEMU programs. If |
| 240 | you opt for Python, it is strongly recommended to write Python 3 compatible |
| 241 | code. |
| 242 | |
Fam Zheng | f4a1b65 | 2018-07-24 16:47:38 +0800 | [diff] [blame] | 243 | Both Python and Bash frameworks in iotests provide helpers to manage test |
| 244 | images. They can be used to create and clean up images under the test |
| 245 | directory. If no I/O or any protocol specific feature is needed, it is often |
| 246 | more convenient to use the pseudo block driver, ``null-co://``, as the test |
| 247 | image, which doesn't require image creation or cleaning up. Avoid system-wide |
| 248 | devices or files whenever possible, such as ``/dev/null`` or ``/dev/zero``. |
| 249 | Otherwise, image locking implications have to be considered. For example, |
| 250 | another application on the host may have locked the file, possibly leading to a |
| 251 | test failure. If using such devices are explicitly desired, consider adding |
| 252 | ``locking=off`` option to disable image locking. |
| 253 | |
Emanuele Giuseppe Esposito | 0193767 | 2021-08-09 11:01:02 +0200 | [diff] [blame] | 254 | Debugging a test case |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 255 | ~~~~~~~~~~~~~~~~~~~~~ |
| 256 | |
Emanuele Giuseppe Esposito | 0193767 | 2021-08-09 11:01:02 +0200 | [diff] [blame] | 257 | The following options to the ``check`` script can be useful when debugging |
| 258 | a failing test: |
| 259 | |
Emanuele Giuseppe Esposito | e92ecc3 | 2021-08-09 11:01:07 +0200 | [diff] [blame] | 260 | * ``-gdb`` wraps every QEMU invocation in a ``gdbserver``, which waits for a |
| 261 | connection from a gdb client. The options given to ``gdbserver`` (e.g. the |
| 262 | address on which to listen for connections) are taken from the ``$GDB_OPTIONS`` |
| 263 | environment variable. By default (if ``$GDB_OPTIONS`` is empty), it listens on |
| 264 | ``localhost:12345``. |
| 265 | It is possible to connect to it for example with |
| 266 | ``gdb -iex "target remote $addr"``, where ``$addr`` is the address |
| 267 | ``gdbserver`` listens on. |
| 268 | If the ``-gdb`` option is not used, ``$GDB_OPTIONS`` is ignored, |
| 269 | regardless of whether it is set or not. |
| 270 | |
Emanuele Giuseppe Esposito | bd10a73 | 2021-08-09 11:01:12 +0200 | [diff] [blame] | 271 | * ``-valgrind`` attaches a valgrind instance to QEMU. If it detects |
| 272 | warnings, it will print and save the log in |
| 273 | ``$TEST_DIR/<valgrind_pid>.valgrind``. |
| 274 | The final command line will be ``valgrind --log-file=$TEST_DIR/ |
| 275 | <valgrind_pid>.valgrind --error-exitcode=99 $QEMU ...`` |
| 276 | |
Emanuele Giuseppe Esposito | 0193767 | 2021-08-09 11:01:02 +0200 | [diff] [blame] | 277 | * ``-d`` (debug) just increases the logging verbosity, showing |
| 278 | for example the QMP commands and answers. |
| 279 | |
Emanuele Giuseppe Esposito | 8ffcda2 | 2021-08-09 11:01:14 +0200 | [diff] [blame] | 280 | * ``-p`` (print) redirects QEMU’s stdout and stderr to the test output, |
| 281 | instead of saving it into a log file in |
| 282 | ``$TEST_DIR/qemu-machine-<random_string>``. |
| 283 | |
Vladimir Sementsov-Ogievskiy | b25a948 | 2021-01-25 21:50:52 +0300 | [diff] [blame] | 284 | Test case groups |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 285 | ~~~~~~~~~~~~~~~~ |
Vladimir Sementsov-Ogievskiy | b25a948 | 2021-01-25 21:50:52 +0300 | [diff] [blame] | 286 | |
| 287 | "Tests may belong to one or more test groups, which are defined in the form |
| 288 | of a comment in the test source file. By convention, test groups are listed |
| 289 | in the second line of the test file, after the "#!/..." line, like this: |
| 290 | |
| 291 | .. code:: |
| 292 | |
| 293 | #!/usr/bin/env python3 |
| 294 | # group: auto quick |
| 295 | # |
| 296 | ... |
| 297 | |
| 298 | Another way of defining groups is creating the tests/qemu-iotests/group.local |
| 299 | file. This should be used only for downstream (this file should never appear |
| 300 | in upstream). This file may be used for defining some downstream test groups |
| 301 | or for temporarily disabling tests, like this: |
| 302 | |
| 303 | .. code:: |
| 304 | |
| 305 | # groups for some company downstream process |
| 306 | # |
| 307 | # ci - tests to run on build |
| 308 | # down - our downstream tests, not for upstream |
| 309 | # |
| 310 | # Format of each line is: |
| 311 | # TEST_NAME TEST_GROUP [TEST_GROUP ]... |
| 312 | |
| 313 | 013 ci |
| 314 | 210 disabled |
| 315 | 215 disabled |
| 316 | our-ugly-workaround-test down ci |
| 317 | |
| 318 | Note that the following group names have a special meaning: |
| 319 | |
| 320 | - quick: Tests in this group should finish within a few seconds. |
| 321 | |
| 322 | - auto: Tests in this group are used during "make check" and should be |
| 323 | runnable in any case. That means they should run with every QEMU binary |
| 324 | (also non-x86), with every QEMU configuration (i.e. must not fail if |
| 325 | an optional feature is not compiled in - but reporting a "skip" is ok), |
| 326 | work at least with the qcow2 file format, work with all kind of host |
| 327 | filesystems and users (e.g. "nobody" or "root") and must not take too |
| 328 | much memory and disk space (since CI pipelines tend to fail otherwise). |
| 329 | |
| 330 | - disabled: Tests in this group are disabled and ignored by check. |
| 331 | |
Alex Bennée | 663a041 | 2021-02-22 10:14:53 +0000 | [diff] [blame] | 332 | .. _container-ref: |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 333 | |
Alex Bennée | 663a041 | 2021-02-22 10:14:53 +0000 | [diff] [blame] | 334 | Container based tests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 335 | --------------------- |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 336 | |
| 337 | Introduction |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 338 | ~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 339 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 340 | The container testing framework in QEMU utilizes public images to |
| 341 | build and test QEMU in predefined and widely accessible Linux |
| 342 | environments. This makes it possible to expand the test coverage |
| 343 | across distros, toolchain flavors and library versions. The support |
| 344 | was originally written for Docker although we also support Podman as |
Matheus Tavares Bernardino | 0285a0e | 2022-08-23 10:46:19 -0300 | [diff] [blame] | 345 | an alternative container runtime. Although many of the target |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 346 | names and scripts are prefixed with "docker" the system will |
| 347 | automatically run on whichever is configured. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 348 | |
Alex Bennée | 4583cda | 2021-02-22 10:14:55 +0000 | [diff] [blame] | 349 | The container images are also used to augment the generation of tests |
| 350 | for testing TCG. See :ref:`checktcg-ref` for more details. |
| 351 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 352 | Docker Prerequisites |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 353 | ~~~~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 354 | |
| 355 | Install "docker" with the system package manager and start the Docker service |
| 356 | on your development machine, then make sure you have the privilege to run |
| 357 | Docker commands. Typically it means setting up passwordless ``sudo docker`` |
| 358 | command or login as root. For example: |
| 359 | |
| 360 | .. code:: |
| 361 | |
| 362 | $ sudo yum install docker |
| 363 | $ # or `apt-get install docker` for Ubuntu, etc. |
| 364 | $ sudo systemctl start docker |
| 365 | $ sudo docker ps |
| 366 | |
| 367 | The last command should print an empty table, to verify the system is ready. |
| 368 | |
| 369 | An alternative method to set up permissions is by adding the current user to |
| 370 | "docker" group and making the docker daemon socket file (by default |
| 371 | ``/var/run/docker.sock``) accessible to the group: |
| 372 | |
| 373 | .. code:: |
| 374 | |
| 375 | $ sudo groupadd docker |
Murilo Opsfelder Araujo | 29c33cc | 2019-02-07 16:43:46 -0200 | [diff] [blame] | 376 | $ sudo usermod $USER -a -G docker |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 377 | $ sudo chown :docker /var/run/docker.sock |
| 378 | |
| 379 | Note that any one of above configurations makes it possible for the user to |
| 380 | exploit the whole host with Docker bind mounting or other privileged |
| 381 | operations. So only do it on development machines. |
| 382 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 383 | Podman Prerequisites |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 384 | ~~~~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 385 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 386 | Install "podman" with the system package manager. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 387 | |
| 388 | .. code:: |
| 389 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 390 | $ sudo dnf install podman |
| 391 | $ podman ps |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 392 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 393 | The last command should print an empty table, to verify the system is ready. |
| 394 | |
| 395 | Quickstart |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 396 | ~~~~~~~~~~ |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 397 | |
| 398 | From source tree, type ``make docker-help`` to see the help. Testing |
| 399 | can be started without configuring or building QEMU (``configure`` and |
| 400 | ``make`` are done in the container, with parameters defined by the |
| 401 | make target): |
| 402 | |
| 403 | .. code:: |
| 404 | |
Alex Bennée | cc1d2e0 | 2024-06-03 18:53:18 +0100 | [diff] [blame] | 405 | make docker-test-build@debian |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 406 | |
Alex Bennée | cc1d2e0 | 2024-06-03 18:53:18 +0100 | [diff] [blame] | 407 | This will create a container instance using the ``debian`` image (the image |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 408 | is downloaded and initialized automatically), in which the ``test-build`` job |
| 409 | is executed. |
| 410 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 411 | Registry |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 412 | ~~~~~~~~ |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 413 | |
| 414 | The QEMU project has a container registry hosted by GitLab at |
| 415 | ``registry.gitlab.com/qemu-project/qemu`` which will automatically be |
| 416 | used to pull in pre-built layers. This avoids unnecessary strain on |
| 417 | the distro archives created by multiple developers running the same |
| 418 | container build steps over and over again. This can be overridden |
| 419 | locally by using the ``NOCACHE`` build option: |
| 420 | |
| 421 | .. code:: |
| 422 | |
Alex Bennée | d996f0a | 2022-09-14 16:59:50 +0100 | [diff] [blame] | 423 | make docker-image-debian-arm64-cross NOCACHE=1 |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 424 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 425 | Images |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 426 | ~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 427 | |
Alex Bennée | cc1d2e0 | 2024-06-03 18:53:18 +0100 | [diff] [blame] | 428 | Along with many other images, the ``debian`` image is defined in a Dockerfile |
| 429 | in ``tests/docker/dockerfiles/``, called ``debian.docker``. ``make docker-help`` |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 430 | command will list all the available images. |
| 431 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 432 | A ``.pre`` script can be added beside the ``.docker`` file, which will be |
| 433 | executed before building the image under the build context directory. This is |
| 434 | mainly used to do necessary host side setup. One such setup is ``binfmt_misc``, |
| 435 | for example, to make qemu-user powered cross build containers work. |
| 436 | |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 437 | Most of the existing Dockerfiles were written by hand, simply by creating a |
| 438 | a new ``.docker`` file under the ``tests/docker/dockerfiles/`` directory. |
| 439 | This has led to an inconsistent set of packages being present across the |
| 440 | different containers. |
| 441 | |
| 442 | Thus going forward, QEMU is aiming to automatically generate the Dockerfiles |
| 443 | using the ``lcitool`` program provided by the ``libvirt-ci`` project: |
| 444 | |
| 445 | https://gitlab.com/libvirt/libvirt-ci |
| 446 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 447 | ``libvirt-ci`` contains an ``lcitool`` program as well as a list of |
| 448 | mappings to distribution package names for a wide variety of third |
| 449 | party projects. ``lcitool`` applies the mappings to a list of build |
| 450 | pre-requisites in ``tests/lcitool/projects/qemu.yml``, determines the |
| 451 | list of native packages to install on each distribution, and uses them |
| 452 | to generate build environments (dockerfiles and Cirrus CI variable files) |
| 453 | that are consistent across OS distribution. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 454 | |
| 455 | |
| 456 | Adding new build pre-requisites |
| 457 | ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 458 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 459 | When preparing a patch series that adds a new build |
| 460 | pre-requisite to QEMU, the prerequisites should to be added to |
| 461 | ``tests/lcitool/projects/qemu.yml`` in order to make the dependency |
| 462 | available in the CI build environments. |
| 463 | |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 464 | In the simple case where the pre-requisite is already known to ``libvirt-ci`` |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 465 | the following steps are needed: |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 466 | |
| 467 | * Edit ``tests/lcitool/projects/qemu.yml`` and add the pre-requisite |
| 468 | |
| 469 | * Run ``make lcitool-refresh`` to re-generate all relevant build environment |
| 470 | manifests |
| 471 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 472 | It may be that ``libvirt-ci`` does not know about the new pre-requisite. |
| 473 | If that is the case, some extra preparation steps will be required |
| 474 | first to contribute the mapping to the ``libvirt-ci`` project: |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 475 | |
| 476 | * Fork the ``libvirt-ci`` project on gitlab |
| 477 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 478 | * Add an entry for the new build prerequisite to |
| 479 | ``lcitool/facts/mappings.yml``, listing its native package name on as |
| 480 | many OS distros as practical. Run ``python -m pytest --regenerate-output`` |
| 481 | and check that the changes are correct. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 482 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 483 | * Commit the ``mappings.yml`` change together with the regenerated test |
| 484 | files, and submit a merge request to the ``libvirt-ci`` project. |
| 485 | Please note in the description that this is a new build pre-requisite |
| 486 | desired for use with QEMU. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 487 | |
| 488 | * CI pipeline will run to validate that the changes to ``mappings.yml`` |
| 489 | are correct, by attempting to install the newly listed package on |
| 490 | all OS distributions supported by ``libvirt-ci``. |
| 491 | |
| 492 | * Once the merge request is accepted, go back to QEMU and update |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 493 | the ``tests/lcitool/libvirt-ci`` submodule to point to a commit that |
| 494 | contains the ``mappings.yml`` update. Then add the prerequisite and |
| 495 | run ``make lcitool-refresh``. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 496 | |
Ani Sinha | 2a851fc | 2023-05-06 12:50:12 +0530 | [diff] [blame] | 497 | * Please also trigger gitlab container generation pipelines on your change |
| 498 | for as many OS distros as practical to make sure that there are no |
| 499 | obvious breakages when adding the new pre-requisite. Please see |
| 500 | `CI <https://www.qemu.org/docs/master/devel/ci.html>`__ documentation |
| 501 | page on how to trigger gitlab CI pipelines on your change. |
| 502 | |
Paolo Bonzini | 32c0613 | 2022-10-12 18:20:44 +0200 | [diff] [blame] | 503 | For enterprise distros that default to old, end-of-life versions of the |
| 504 | Python runtime, QEMU uses a separate set of mappings that work with more |
| 505 | recent versions. These can be found in ``tests/lcitool/mappings.yml``. |
| 506 | Modifying this file should not be necessary unless the new pre-requisite |
| 507 | is a Python library or tool. |
| 508 | |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 509 | |
| 510 | Adding new OS distros |
| 511 | ^^^^^^^^^^^^^^^^^^^^^ |
| 512 | |
| 513 | In some cases ``libvirt-ci`` will not know about the OS distro that is |
| 514 | desired to be tested. Before adding a new OS distro, discuss the proposed |
| 515 | addition: |
| 516 | |
| 517 | * Send a mail to qemu-devel, copying people listed in the |
| 518 | MAINTAINERS file for ``Build and test automation``. |
| 519 | |
| 520 | There are limited CI compute resources available to QEMU, so the |
| 521 | cost/benefit tradeoff of adding new OS distros needs to be considered. |
| 522 | |
| 523 | * File an issue at https://gitlab.com/libvirt/libvirt-ci/-/issues |
| 524 | pointing to the qemu-devel mail thread in the archives. |
| 525 | |
| 526 | This alerts other people who might be interested in the work |
| 527 | to avoid duplication, as well as to get feedback from libvirt-ci |
| 528 | maintainers on any tips to ease the addition |
| 529 | |
| 530 | Assuming there is agreement to add a new OS distro then |
| 531 | |
| 532 | * Fork the ``libvirt-ci`` project on gitlab |
| 533 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 534 | * Add metadata under ``lcitool/facts/targets/`` for the new OS |
| 535 | distro. There might be code changes required if the OS distro |
| 536 | uses a package format not currently known. The ``libvirt-ci`` |
| 537 | maintainers can advise on this when the issue is filed. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 538 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 539 | * Edit the ``lcitool/facts/mappings.yml`` change to add entries for |
| 540 | the new OS, listing the native package names for as many packages |
| 541 | as practical. Run ``python -m pytest --regenerate-output`` and |
| 542 | check that the changes are correct. |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 543 | |
Paolo Bonzini | fa1ce1d | 2023-02-09 12:24:26 +0100 | [diff] [blame] | 544 | * Commit the changes to ``lcitool/facts`` and the regenerated test |
| 545 | files, and submit a merge request to the ``libvirt-ci`` project. |
| 546 | Please note in the description that this is a new build pre-requisite |
| 547 | desired for use with QEMU |
Daniel P. Berrangé | 4ebb040 | 2022-01-05 13:49:42 +0000 | [diff] [blame] | 548 | |
| 549 | * CI pipeline will run to validate that the changes to ``mappings.yml`` |
| 550 | are correct, by attempting to install the newly listed package on |
| 551 | all OS distributions supported by ``libvirt-ci``. |
| 552 | |
| 553 | * Once the merge request is accepted, go back to QEMU and update |
| 554 | the ``libvirt-ci`` submodule to point to a commit that contains |
| 555 | the ``mappings.yml`` update. |
| 556 | |
| 557 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 558 | Tests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 559 | ~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 560 | |
| 561 | Different tests are added to cover various configurations to build and test |
| 562 | QEMU. Docker tests are the executables under ``tests/docker`` named |
| 563 | ``test-*``. They are typically shell scripts and are built on top of a shell |
| 564 | library, ``tests/docker/common.rc``, which provides helpers to find the QEMU |
| 565 | source and build it. |
| 566 | |
Alex Bennée | 9c1f491 | 2021-02-22 10:14:54 +0000 | [diff] [blame] | 567 | The full list of tests is printed in the ``make docker-help`` help. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 568 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 569 | Debugging a Docker test failure |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 570 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 571 | |
| 572 | When CI tasks, maintainers or yourself report a Docker test failure, follow the |
| 573 | below steps to debug it: |
| 574 | |
| 575 | 1. Locally reproduce the failure with the reported command line. E.g. run |
Philippe Mathieu-Daudé | 098bfc0 | 2023-06-24 14:58:24 +0200 | [diff] [blame] | 576 | ``make docker-test-mingw@fedora-win64-cross J=8``. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 577 | 2. Add "V=1" to the command line, try again, to see the verbose output. |
| 578 | 3. Further add "DEBUG=1" to the command line. This will pause in a shell prompt |
| 579 | in the container right before testing starts. You could either manually |
| 580 | build QEMU and run tests from there, or press Ctrl-D to let the Docker |
| 581 | testing continue. |
| 582 | 4. If you press Ctrl-D, the same building and testing procedure will begin, and |
| 583 | will hopefully run into the error again. After that, you will be dropped to |
| 584 | the prompt for debug. |
| 585 | |
| 586 | Options |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 587 | ~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 588 | |
| 589 | Various options can be used to affect how Docker tests are done. The full |
| 590 | list is in the ``make docker`` help text. The frequently used ones are: |
| 591 | |
| 592 | * ``V=1``: the same as in top level ``make``. It will be propagated to the |
| 593 | container and enable verbose output. |
| 594 | * ``J=$N``: the number of parallel tasks in make commands in the container, |
| 595 | similar to the ``-j $N`` option in top level ``make``. (The ``-j`` option in |
| 596 | top level ``make`` will not be propagated into the container.) |
| 597 | * ``DEBUG=1``: enables debug. See the previous "Debugging a Docker test |
| 598 | failure" section. |
| 599 | |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 600 | Thread Sanitizer |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 601 | ---------------- |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 602 | |
| 603 | Thread Sanitizer (TSan) is a tool which can detect data races. QEMU supports |
| 604 | building and testing with this tool. |
| 605 | |
| 606 | For more information on TSan: |
| 607 | |
| 608 | https://github.com/google/sanitizers/wiki/ThreadSanitizerCppManual |
| 609 | |
| 610 | Thread Sanitizer in Docker |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 611 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Alex Bennée | 171080d | 2023-02-28 19:06:37 +0000 | [diff] [blame] | 612 | TSan is currently supported in the ubuntu2204 docker. |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 613 | |
| 614 | The test-tsan test will build using TSan and then run make check. |
| 615 | |
| 616 | .. code:: |
| 617 | |
Alex Bennée | 171080d | 2023-02-28 19:06:37 +0000 | [diff] [blame] | 618 | make docker-test-tsan@ubuntu2204 |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 619 | |
| 620 | TSan warnings under docker are placed in files located at build/tsan/. |
| 621 | |
| 622 | We recommend using DEBUG=1 to allow launching the test from inside the docker, |
| 623 | and to allow review of the warnings generated by TSan. |
| 624 | |
| 625 | Building and Testing with TSan |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 626 | ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 627 | |
| 628 | It is possible to build and test with TSan, with a few additional steps. |
| 629 | These steps are normally done automatically in the docker. |
| 630 | |
| 631 | There is a one time patch needed in clang-9 or clang-10 at this time: |
| 632 | |
| 633 | .. code:: |
| 634 | |
| 635 | sed -i 's/^const/static const/g' \ |
| 636 | /usr/lib/llvm-10/lib/clang/10.0.0/include/sanitizer/tsan_interface.h |
| 637 | |
| 638 | To configure the build for TSan: |
| 639 | |
| 640 | .. code:: |
| 641 | |
| 642 | ../configure --enable-tsan --cc=clang-10 --cxx=clang++-10 \ |
| 643 | --disable-werror --extra-cflags="-O0" |
| 644 | |
| 645 | The runtime behavior of TSAN is controlled by the TSAN_OPTIONS environment |
| 646 | variable. |
| 647 | |
| 648 | More information on the TSAN_OPTIONS can be found here: |
| 649 | |
| 650 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags |
| 651 | |
| 652 | For example: |
| 653 | |
| 654 | .. code:: |
| 655 | |
| 656 | export TSAN_OPTIONS=suppressions=<path to qemu>/tests/tsan/suppressions.tsan \ |
| 657 | detect_deadlocks=false history_size=7 exitcode=0 \ |
| 658 | log_path=<build path>/tsan/tsan_warning |
| 659 | |
| 660 | The above exitcode=0 has TSan continue without error if any warnings are found. |
| 661 | This allows for running the test and then checking the warnings afterwards. |
| 662 | If you want TSan to stop and exit with error on warnings, use exitcode=66. |
| 663 | |
| 664 | TSan Suppressions |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 665 | ~~~~~~~~~~~~~~~~~ |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 666 | Keep in mind that for any data race warning, although there might be a data race |
| 667 | detected by TSan, there might be no actual bug here. TSan provides several |
| 668 | different mechanisms for suppressing warnings. In general it is recommended |
| 669 | to fix the code if possible to eliminate the data race rather than suppress |
| 670 | the warning. |
| 671 | |
| 672 | A few important files for suppressing warnings are: |
| 673 | |
| 674 | tests/tsan/suppressions.tsan - Has TSan warnings we wish to suppress at runtime. |
zhaolichang | 76ca4b5 | 2020-09-17 15:50:22 +0800 | [diff] [blame] | 675 | The comment on each suppression will typically indicate why we are |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 676 | suppressing it. More information on the file format can be found here: |
| 677 | |
| 678 | https://github.com/google/sanitizers/wiki/ThreadSanitizerSuppressions |
| 679 | |
Thomas Huth | f9a19bd | 2023-11-09 18:47:20 +0100 | [diff] [blame] | 680 | tests/tsan/ignore.tsan - Has TSan warnings we wish to disable |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 681 | at compile time for test or debug. |
| 682 | Add flags to configure to enable: |
| 683 | |
Thomas Huth | f9a19bd | 2023-11-09 18:47:20 +0100 | [diff] [blame] | 684 | "--extra-cflags=-fsanitize-blacklist=<src path>/tests/tsan/ignore.tsan" |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 685 | |
| 686 | More information on the file format can be found here under "Blacklist Format": |
| 687 | |
| 688 | https://github.com/google/sanitizers/wiki/ThreadSanitizerFlags |
| 689 | |
| 690 | TSan Annotations |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 691 | ~~~~~~~~~~~~~~~~ |
Robert Foley | 3b6882b | 2020-06-12 20:02:34 +0100 | [diff] [blame] | 692 | include/qemu/tsan.h defines annotations. See this file for more descriptions |
| 693 | of the annotations themselves. Annotations can be used to suppress |
| 694 | TSan warnings or give TSan more information so that it can detect proper |
| 695 | relationships between accesses of data. |
| 696 | |
| 697 | Annotation examples can be found here: |
| 698 | |
| 699 | https://github.com/llvm/llvm-project/tree/master/compiler-rt/test/tsan/ |
| 700 | |
| 701 | Good files to start with are: annotate_happens_before.cpp and ignore_race.cpp |
| 702 | |
| 703 | The full set of annotations can be found here: |
| 704 | |
| 705 | https://github.com/llvm/llvm-project/blob/master/compiler-rt/lib/tsan/rtl/tsan_interface_ann.cpp |
| 706 | |
Alex Bennée | 396408e | 2022-04-19 10:09:57 +0100 | [diff] [blame] | 707 | docker-binfmt-image-debian-% targets |
| 708 | ------------------------------------ |
| 709 | |
| 710 | It is possible to combine Debian's bootstrap scripts with a configured |
| 711 | ``binfmt_misc`` to bootstrap a number of Debian's distros including |
| 712 | experimental ports not yet supported by a released OS. This can |
| 713 | simplify setting up a rootfs by using docker to contain the foreign |
| 714 | rootfs rather than manually invoking chroot. |
| 715 | |
| 716 | Setting up ``binfmt_misc`` |
| 717 | ~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 718 | |
| 719 | You can use the script ``qemu-binfmt-conf.sh`` to configure a QEMU |
| 720 | user binary to automatically run binaries for the foreign |
| 721 | architecture. While the scripts will try their best to work with |
| 722 | dynamically linked QEMU's a statically linked one will present less |
| 723 | potential complications when copying into the docker image. Modern |
| 724 | kernels support the ``F`` (fix binary) flag which will open the QEMU |
| 725 | executable on setup and avoids the need to find and re-open in the |
| 726 | chroot environment. This is triggered with the ``--persistent`` flag. |
| 727 | |
| 728 | Example invocation |
| 729 | ~~~~~~~~~~~~~~~~~~ |
| 730 | |
| 731 | For example to setup the HPPA ports builds of Debian:: |
| 732 | |
| 733 | make docker-binfmt-image-debian-sid-hppa \ |
| 734 | DEB_TYPE=sid DEB_ARCH=hppa \ |
| 735 | DEB_URL=http://ftp.ports.debian.org/debian-ports/ \ |
| 736 | DEB_KEYRING=/usr/share/keyrings/debian-ports-archive-keyring.gpg \ |
| 737 | EXECUTABLE=(pwd)/qemu-hppa V=1 |
| 738 | |
| 739 | The ``DEB_`` variables are substitutions used by |
Manos Pitsidianakis | c555b50 | 2024-02-20 10:52:08 +0200 | [diff] [blame] | 740 | ``debian-bootstrap.pre`` which is called to do the initial debootstrap |
Alex Bennée | 396408e | 2022-04-19 10:09:57 +0100 | [diff] [blame] | 741 | of the rootfs before it is copied into the container. The second stage |
| 742 | is run as part of the build. The final image will be tagged as |
| 743 | ``qemu/debian-sid-hppa``. |
| 744 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 745 | VM testing |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 746 | ---------- |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 747 | |
| 748 | This test suite contains scripts that bootstrap various guest images that have |
| 749 | necessary packages to build QEMU. The basic usage is documented in ``Makefile`` |
Philippe Mathieu-Daudé | 4f2f627 | 2019-05-31 08:43:41 +0200 | [diff] [blame] | 750 | help which is displayed with ``make vm-help``. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 751 | |
| 752 | Quickstart |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 753 | ~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 754 | |
Philippe Mathieu-Daudé | 4f2f627 | 2019-05-31 08:43:41 +0200 | [diff] [blame] | 755 | Run ``make vm-help`` to list available make targets. Invoke a specific make |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 756 | command to run build test in an image. For example, ``make vm-build-freebsd`` |
| 757 | will build the source tree in the FreeBSD image. The command can be executed |
| 758 | from either the source tree or the build dir; if the former, ``./configure`` is |
| 759 | not needed. The command will then generate the test image in ``./tests/vm/`` |
| 760 | under the working directory. |
| 761 | |
| 762 | Note: images created by the scripts accept a well-known RSA key pair for SSH |
| 763 | access, so they SHOULD NOT be exposed to external interfaces if you are |
| 764 | concerned about attackers taking control of the guest and potentially |
| 765 | exploiting a QEMU security bug to compromise the host. |
| 766 | |
Wainer dos Santos Moschetta | 1e48931 | 2019-11-14 08:42:46 -0500 | [diff] [blame] | 767 | QEMU binaries |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 768 | ~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 769 | |
Philippe Mathieu-Daudé | c5ba621 | 2021-11-18 20:27:44 +0100 | [diff] [blame] | 770 | By default, ``qemu-system-x86_64`` is searched in $PATH to run the guest. If |
| 771 | there isn't one, or if it is older than 2.10, the test won't work. In this case, |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 772 | provide the QEMU binary in env var: ``QEMU=/path/to/qemu-2.10+``. |
| 773 | |
Philippe Mathieu-Daudé | c5ba621 | 2021-11-18 20:27:44 +0100 | [diff] [blame] | 774 | Likewise the path to ``qemu-img`` can be set in QEMU_IMG environment variable. |
Wainer dos Santos Moschetta | 1e48931 | 2019-11-14 08:42:46 -0500 | [diff] [blame] | 775 | |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 776 | Make jobs |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 777 | ~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 778 | |
| 779 | The ``-j$X`` option in the make command line is not propagated into the VM, |
| 780 | specify ``J=$X`` to control the make jobs in the guest. |
| 781 | |
| 782 | Debugging |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 783 | ~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 784 | |
| 785 | Add ``DEBUG=1`` and/or ``V=1`` to the make command to allow interactive |
| 786 | debugging and verbose output. If this is not enough, see the next section. |
Peter Maydell | 41e3340 | 2018-08-03 09:52:28 +0100 | [diff] [blame] | 787 | ``V=1`` will be propagated down into the make jobs in the guest. |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 788 | |
| 789 | Manual invocation |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 790 | ~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 791 | |
| 792 | Each guest script is an executable script with the same command line options. |
| 793 | For example to work with the netbsd guest, use ``$QEMU_SRC/tests/vm/netbsd``: |
| 794 | |
| 795 | .. code:: |
| 796 | |
| 797 | $ cd $QEMU_SRC/tests/vm |
| 798 | |
| 799 | # To bootstrap the image |
| 800 | $ ./netbsd --build-image --image /var/tmp/netbsd.img |
| 801 | <...> |
| 802 | |
| 803 | # To run an arbitrary command in guest (the output will not be echoed unless |
| 804 | # --debug is added) |
| 805 | $ ./netbsd --debug --image /var/tmp/netbsd.img uname -a |
| 806 | |
| 807 | # To build QEMU in guest |
| 808 | $ ./netbsd --debug --image /var/tmp/netbsd.img --build-qemu $QEMU_SRC |
| 809 | |
| 810 | # To get to an interactive shell |
| 811 | $ ./netbsd --interactive --image /var/tmp/netbsd.img sh |
| 812 | |
| 813 | Adding new guests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 814 | ~~~~~~~~~~~~~~~~~ |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 815 | |
| 816 | Please look at existing guest scripts for how to add new guests. |
| 817 | |
| 818 | Most importantly, create a subclass of BaseVM and implement ``build_image()`` |
| 819 | method and define ``BUILD_SCRIPT``, then finally call ``basevm.main()`` from |
| 820 | the script's ``main()``. |
| 821 | |
| 822 | * Usually in ``build_image()``, a template image is downloaded from a |
| 823 | predefined URL. ``BaseVM._download_with_cache()`` takes care of the cache and |
| 824 | the checksum, so consider using it. |
| 825 | |
| 826 | * Once the image is downloaded, users, SSH server and QEMU build deps should |
| 827 | be set up: |
| 828 | |
| 829 | - Root password set to ``BaseVM.ROOT_PASS`` |
| 830 | - User ``BaseVM.GUEST_USER`` is created, and password set to |
| 831 | ``BaseVM.GUEST_PASS`` |
| 832 | - SSH service is enabled and started on boot, |
| 833 | ``$QEMU_SRC/tests/keys/id_rsa.pub`` is added to ssh's ``authorized_keys`` |
| 834 | file of both root and the normal user |
| 835 | - DHCP client service is enabled and started on boot, so that it can |
| 836 | automatically configure the virtio-net-pci NIC and communicate with QEMU |
| 837 | user net (10.0.2.2) |
| 838 | - Necessary packages are installed to untar the source tarball and build |
| 839 | QEMU |
| 840 | |
| 841 | * Write a proper ``BUILD_SCRIPT`` template, which should be a shell script that |
| 842 | untars a raw virtio-blk block device, which is the tarball data blob of the |
| 843 | QEMU source tree, then configure/build it. Running "make check" is also |
| 844 | recommended. |
| 845 | |
| 846 | Image fuzzer testing |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 847 | -------------------- |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 848 | |
| 849 | An image fuzzer was added to exercise format drivers. Currently only qcow2 is |
| 850 | supported. To start the fuzzer, run |
| 851 | |
| 852 | .. code:: |
| 853 | |
| 854 | tests/image-fuzzer/runner.py -c '[["qemu-img", "info", "$test_img"]]' /tmp/test qcow2 |
| 855 | |
Philippe Mathieu-Daudé | c5ba621 | 2021-11-18 20:27:44 +0100 | [diff] [blame] | 856 | Alternatively, some command different from ``qemu-img info`` can be tested, by |
Fam Zheng | 4eb9956 | 2018-02-01 10:20:46 +0800 | [diff] [blame] | 857 | changing the ``-c`` option. |
Cleber Rosa | c3d7e8c | 2018-05-30 14:41:52 -0400 | [diff] [blame] | 858 | |
Thomas Huth | c3e24cf | 2024-08-30 15:38:38 +0200 | [diff] [blame] | 859 | Functional tests using Python |
| 860 | ----------------------------- |
| 861 | |
| 862 | The ``tests/functional`` directory hosts functional tests written in |
| 863 | Python. You can run the functional tests simply by executing: |
| 864 | |
| 865 | .. code:: |
| 866 | |
| 867 | make check-functional |
| 868 | |
| 869 | See :ref:`checkfunctional-ref` for more details. |
| 870 | |
Willian Rampazzo | bbbd9b6 | 2021-11-05 12:53:54 -0300 | [diff] [blame] | 871 | Integration tests using the Avocado Framework |
| 872 | --------------------------------------------- |
Cleber Rosa | c3d7e8c | 2018-05-30 14:41:52 -0400 | [diff] [blame] | 873 | |
Willian Rampazzo | bbbd9b6 | 2021-11-05 12:53:54 -0300 | [diff] [blame] | 874 | The ``tests/avocado`` directory hosts integration tests. They're usually |
| 875 | higher level tests, and may interact with external resources and with |
| 876 | various guest operating systems. |
Cleber Rosa | c3d7e8c | 2018-05-30 14:41:52 -0400 | [diff] [blame] | 877 | |
Willian Rampazzo | bbbd9b6 | 2021-11-05 12:53:54 -0300 | [diff] [blame] | 878 | You can run the avocado tests simply by executing: |
Cleber Rosa | a56931e | 2018-10-18 11:31:33 -0400 | [diff] [blame] | 879 | |
| 880 | .. code:: |
| 881 | |
Willian Rampazzo | bbbd9b6 | 2021-11-05 12:53:54 -0300 | [diff] [blame] | 882 | make check-avocado |
Cleber Rosa | a56931e | 2018-10-18 11:31:33 -0400 | [diff] [blame] | 883 | |
Thomas Huth | 2133c2a | 2024-08-30 15:38:36 +0200 | [diff] [blame] | 884 | See :ref:`checkavocado-ref` for more details. |
Cleber Rosa | a56931e | 2018-10-18 11:31:33 -0400 | [diff] [blame] | 885 | |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 886 | |
Alex Bennée | 4583cda | 2021-02-22 10:14:55 +0000 | [diff] [blame] | 887 | .. _checktcg-ref: |
| 888 | |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 889 | Testing with "make check-tcg" |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 890 | ----------------------------- |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 891 | |
| 892 | The check-tcg tests are intended for simple smoke tests of both |
| 893 | linux-user and softmmu TCG functionality. However to build test |
| 894 | programs for guest targets you need to have cross compilers available. |
| 895 | If your distribution supports cross compilers you can do something as |
| 896 | simple as:: |
| 897 | |
| 898 | apt install gcc-aarch64-linux-gnu |
| 899 | |
| 900 | The configure script will automatically pick up their presence. |
| 901 | Sometimes compilers have slightly odd names so the availability of |
| 902 | them can be prompted by passing in the appropriate configure option |
| 903 | for the architecture in question, for example:: |
| 904 | |
| 905 | $(configure) --cross-cc-aarch64=aarch64-cc |
| 906 | |
Matheus Ferst | 479ca4c | 2022-01-20 14:31:41 -0300 | [diff] [blame] | 907 | There is also a ``--cross-cc-cflags-ARCH`` flag in case additional |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 908 | compiler flags are needed to build for a given target. |
| 909 | |
Alex Bennée | 663a041 | 2021-02-22 10:14:53 +0000 | [diff] [blame] | 910 | If you have the ability to run containers as the user the build system |
| 911 | will automatically use them where no system compiler is available. For |
| 912 | architectures where we also support building QEMU we will generally |
| 913 | use the same container to build tests. However there are a number of |
| 914 | additional containers defined that have a minimal cross-build |
| 915 | environment that is only suitable for building test cases. Sometimes |
| 916 | we may use a bleeding edge distribution for compiler features needed |
| 917 | for test cases that aren't yet in the LTS distros we support for QEMU |
| 918 | itself. |
| 919 | |
| 920 | See :ref:`container-ref` for more details. |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 921 | |
| 922 | Running subset of tests |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 923 | ~~~~~~~~~~~~~~~~~~~~~~~ |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 924 | |
| 925 | You can build the tests for one architecture:: |
| 926 | |
| 927 | make build-tcg-tests-$TARGET |
| 928 | |
| 929 | And run with:: |
| 930 | |
| 931 | make run-tcg-tests-$TARGET |
| 932 | |
| 933 | Adding ``V=1`` to the invocation will show the details of how to |
| 934 | invoke QEMU for the test which is useful for debugging tests. |
| 935 | |
Alex Bennée | b421206 | 2024-07-29 15:44:07 +0100 | [diff] [blame] | 936 | Running individual tests |
| 937 | ~~~~~~~~~~~~~~~~~~~~~~~~ |
| 938 | |
| 939 | Tests can also be run directly from the test build directory. If you |
| 940 | run ``make help`` from the test build directory you will get a list of |
| 941 | all the tests that can be run. Please note that same binaries are used |
| 942 | in multiple tests, for example:: |
| 943 | |
| 944 | make run-plugin-test-mmap-with-libinline.so |
| 945 | |
| 946 | will run the mmap test with the ``libinline.so`` TCG plugin. The |
| 947 | gdbstub tests also re-use the test binaries but while exercising gdb. |
| 948 | |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 949 | TCG test dependencies |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 950 | ~~~~~~~~~~~~~~~~~~~~~ |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 951 | |
| 952 | The TCG tests are deliberately very light on dependencies and are |
Philippe Mathieu-Daudé | 7893e42 | 2023-10-04 11:06:20 +0200 | [diff] [blame] | 953 | either totally bare with minimal gcc lib support (for system-mode tests) |
Alex Bennée | f8ed349 | 2019-09-19 14:36:35 +0100 | [diff] [blame] | 954 | or just glibc (for linux-user tests). This is because getting a cross |
| 955 | compiler to work with additional libraries can be challenging. |
| 956 | |
| 957 | Other TCG Tests |
| 958 | --------------- |
| 959 | |
| 960 | There are a number of out-of-tree test suites that are used for more |
| 961 | extensive testing of processor features. |
| 962 | |
| 963 | KVM Unit Tests |
| 964 | ~~~~~~~~~~~~~~ |
| 965 | |
| 966 | The KVM unit tests are designed to run as a Guest OS under KVM but |
| 967 | there is no reason why they can't exercise the TCG as well. It |
| 968 | provides a minimal OS kernel with hooks for enabling the MMU as well |
| 969 | as reporting test results via a special device:: |
| 970 | |
| 971 | https://git.kernel.org/pub/scm/virt/kvm/kvm-unit-tests.git |
| 972 | |
| 973 | Linux Test Project |
| 974 | ~~~~~~~~~~~~~~~~~~ |
| 975 | |
| 976 | The LTP is focused on exercising the syscall interface of a Linux |
| 977 | kernel. It checks that syscalls behave as documented and strives to |
| 978 | exercise as many corner cases as possible. It is a useful test suite |
| 979 | to run to exercise QEMU's linux-user code:: |
| 980 | |
| 981 | https://linux-test-project.github.io/ |
Paolo Bonzini | 9fce360 | 2021-09-22 17:06:57 +0200 | [diff] [blame] | 982 | |
| 983 | GCC gcov support |
Paolo Bonzini | 16e79e1 | 2021-09-07 17:21:58 +0200 | [diff] [blame] | 984 | ---------------- |
Paolo Bonzini | 9fce360 | 2021-09-22 17:06:57 +0200 | [diff] [blame] | 985 | |
| 986 | ``gcov`` is a GCC tool to analyze the testing coverage by |
| 987 | instrumenting the tested code. To use it, configure QEMU with |
| 988 | ``--enable-gcov`` option and build. Then run the tests as usual. |
| 989 | |
| 990 | If you want to gather coverage information on a single test the ``make |
| 991 | clean-gcda`` target can be used to delete any existing coverage |
| 992 | information before running a single test. |
| 993 | |
| 994 | You can generate a HTML coverage report by executing ``make |
| 995 | coverage-html`` which will create |
| 996 | ``meson-logs/coveragereport/index.html``. |
| 997 | |
| 998 | Further analysis can be conducted by running the ``gcov`` command |
| 999 | directly on the various .gcda output files. Please read the ``gcov`` |
| 1000 | documentation for more information. |