    The QEMU build system architecture
    ==================================

This document aims to help developers understand the architecture of the
QEMU build system. As with projects using GNU autotools, the QEMU build
system has two stages, first the developer runs the "configure" script
to determine the local build environment characteristics, then they run
"make" to build the project. There is about where the similarities with
GNU autotools end, so try to forget what you know about them.


Stage 1: configure
==================

The QEMU configure script is written directly in shell, and should be
compatible with any POSIX shell, hence it uses #!/bin/sh. An important
implication of this is that it is important to avoid using bash-isms on
development platforms where bash is the primary host.

In contrast to autoconf scripts, QEMU's configure is expected to be
silent while it is checking for features. It will only display output
when an error occurs, or to show the final feature enablement summary
on completion.

Adding new checks to the configure script usually comprises the
following tasks:

 - Initialize one or more variables with the default feature state.

   Ideally features should auto-detect whether they are present,
   so try to avoid hardcoding the initial state to either enabled
   or disabled, as that forces the user to pass a --enable-XXX
   / --disable-XXX flag on every invocation of configure.

 - Add support to the command line arg parser to handle any new
   --enable-XXX / --disable-XXX flags required by the feature XXX.

 - Add information to the help output message to report on the new
   feature flag.

 - Add code to perform the actual feature check. As noted above, try to
   be fully dynamic in checking enablement/disablement.

 - Add code to print out the feature status in the configure summary
   upon completion.

 - Add any new makefile variables to $config_host_mak on completion.


Taking (a simplified version of) the probe for gnutls from configure,
we have the following pieces:

  # Initial variable state
  gnutls=""

  ..snip..

  # Configure flag processing
  --disable-gnutls) gnutls="no"
  ;;
  --enable-gnutls) gnutls="yes"
  ;;

  ..snip..

  # Help output feature message
  gnutls          GNUTLS cryptography support

  ..snip..

  # Test for gnutls
  if test "$gnutls" != "no"; then
     if ! $pkg_config --exists "gnutls"; then
        gnutls_cflags=`$pkg_config --cflags gnutls`
        gnutls_libs=`$pkg_config --libs gnutls`
        libs_softmmu="$gnutls_libs $libs_softmmu"
        libs_tools="$gnutls_libs $libs_tools"
        QEMU_CFLAGS="$QEMU_CFLAGS $gnutls_cflags"
        gnutls="yes"
     elif test "$gnutls" = "yes"; then
        feature_not_found "gnutls" "Install gnutls devel"
     else
        gnutls="no"
     fi
  fi

  ..snip..

  # Completion feature summary
  echo "GNUTLS support    $gnutls"

  ..snip..

  # Define make variables
  if test "$gnutls" = "yes" ; then
     echo "CONFIG_GNUTLS=y" >> $config_host_mak
  fi


Helper functions
----------------

The configure script provides a variety of helper functions to assist
developers in checking for system features:

 - do_cc $ARGS...

   Attempt to run the system C compiler passing it $ARGS...

 - do_cxx $ARGS...

   Attempt to run the system C++ compiler passing it $ARGS...

 - compile_object $CFLAGS

   Attempt to compile a test program with the system C compiler using
   $CFLAGS. The test program must have been previously written to a file
   called $TMPC.

 - compile_prog $CFLAGS $LDFLAGS

   Attempt to compile a test program with the system C compiler using
   $CFLAGS and link it with the system linker using $LDFLAGS. The test
   program must have been previously written to a file called $TMPC.

 - has $COMMAND

   Determine if $COMMAND exists in the current environment, either as a
   shell builtin, or executable binary, returning 0 on success.

 - path_of $COMMAND

   Return the fully qualified path of $COMMAND, printing it to stdout,
   and returning 0 on success.

 - check_define $NAME

   Determine if the macro $NAME is defined by the system C compiler

 - check_include $NAME

   Determine if the include $NAME file is available to the system C
   compiler

 - write_c_skeleton

   Write a minimal C program main() function to the temporary file
   indicated by $TMPC

 - feature_not_found $NAME $REMEDY

   Print a message to stderr that the feature $NAME was not available
   on the system, suggesting the user try $REMEDY to address the
   problem.

 - error_exit $MESSAGE $MORE...

   Print $MESSAGE to stderr, followed by $MORE... and then exit from the
   configure script with non-zero status

 - query_pkg_config $ARGS...

   Run pkg-config passing it $ARGS. If QEMU is doing a static build,
   then --static will be automatically added to $ARGS


Stage 2: makefiles
==================

The use of GNU make is required with the QEMU build system.

Although the source code is spread across multiple subdirectories, the
build system should be considered largely non-recursive in nature, in
contrast to common practices seen with automake. There is some recursive
invocation of make, but this is related to the things being built,
rather than the source directory structure.

QEMU currently supports both VPATH and non-VPATH builds, so there are
three general ways to invoke configure & perform a build.

 - VPATH, build artifacts outside of QEMU source tree entirely

     cd ../
     mkdir build
     cd build
     ../qemu/configure
     make

 - VPATH, build artifacts in a subdir of QEMU source tree

     mkdir build
     cd build
     ../configure
     make

 - non-VPATH, build artifacts everywhere

     ./configure
     make

The QEMU maintainers generally recommend that a VPATH build is used by
developers. Patches to QEMU are expected to ensure VPATH build still
works.


Module structure
----------------

There are a number of key outputs of the QEMU build system:

 - Tools - qemu-img, qemu-nbd, qga (guest agent), etc
 - System emulators - qemu-system-$ARCH
 - Userspace emulators - qemu-$ARCH
 - Unit tests

The source code is highly modularized, split across many files to
facilitate building of all of these components with as little duplicated
compilation as possible. There can be considered to be two distinct
groups of files, those which are independent of the QEMU emulation
target and those which are dependent on the QEMU emulation target.

In the target-independent set lives various general purpose helper code,
such as error handling infrastructure, standard data structures,
platform portability wrapper functions, etc. This code can be compiled
once only and the .o files linked into all output binaries.

In the target-dependent set lives CPU emulation, device emulation and
much glue code. This sometimes also has to be compiled multiple times,
once for each target being built.

The utility code that is used by all binaries is built into a
static archive called libqemuutil.a, which is then linked to all the
binaries. In order to provide hooks that are only needed by some of the
binaries, code in libqemuutil.a may depend on other functions that are
not fully implemented by all QEMU binaries.  Dummy stubs for all these
functions are also provided by this library, and will only be linked
into the binary if the real implementation is not present.  In a way,
the stubs can be thought of as a portable implementation of the weak
symbols concept.

All binaries should link to libqemuutil.a, e.g.:

 qemu-img$(EXESUF): qemu-img.o ..snip.. libqemuutil.a


Windows platform portability
----------------------------

On Windows, all binaries have the suffix '.exe', so all Makefile rules
which create binaries must include the $(EXESUF) variable on the binary
name. e.g.

 qemu-img$(EXESUF): qemu-img.o ..snip..

This expands to '.exe' on Windows, or '' on other platforms.

A further complication for the system emulator binaries is that
two separate binaries need to be generated.

The main binary (e.g. qemu-system-x86_64.exe) is linked against the
Windows console runtime subsystem. These are expected to be run from a
command prompt window, and so will print stderr to the console that
launched them.

The second binary generated has a 'w' on the end of its name (e.g.
qemu-system-x86_64w.exe) and is linked against the Windows graphical
runtime subsystem. These are expected to be run directly from the
desktop and will open up a dedicated console window for stderr output.

The Makefile.target will generate the binary for the graphical subsystem
first, and then use objcopy to relink it against the console subsystem
to generate the second binary.


Object variable naming
----------------------

The QEMU convention is to define variables to list different groups of
object files. These are named with the convention $PREFIX-obj-y. For
example the libqemuutil.a file will be linked with all objects listed
in a variable 'util-obj-y'. So, for example, util/Makefile.obj will
contain a set of definitions looking like

  util-obj-y += bitmap.o bitops.o hbitmap.o
  util-obj-y += fifo8.o
  util-obj-y += acl.o
  util-obj-y += error.o qemu-error.o

When there is an object file which needs to be conditionally built based
on some characteristic of the host system, the configure script will
define a variable for the conditional. For example, on Windows it will
define $(CONFIG_POSIX) with a value of 'n' and $(CONFIG_WIN32) with a
value of 'y'. It is now possible to use the config variables when
listing object files. For example,

  util-obj-$(CONFIG_WIN32) += oslib-win32.o qemu-thread-win32.o
  util-obj-$(CONFIG_POSIX) += oslib-posix.o qemu-thread-posix.o

On Windows this expands to

  util-obj-y += oslib-win32.o qemu-thread-win32.o
  util-obj-n += oslib-posix.o qemu-thread-posix.o

Since libqemutil.a links in $(util-obj-y), the POSIX specific files
listed against $(util-obj-n) are ignored on the Windows platform builds.


CFLAGS / LDFLAGS / LIBS handling
--------------------------------

There are many different binaries being built with differing purposes,
and some of them might even be 3rd party libraries pulled in via git
submodules. As such the use of the global CFLAGS variable is generally
avoided in QEMU, since it would apply to too many build targets.

Flags that are needed by any QEMU code (i.e. everything *except* GIT
submodule projects) are put in $(QEMU_CFLAGS) variable. For linker
flags the $(LIBS) variable is sometimes used, but a couple of more
targeted variables are preferred. $(libs_softmmu) is used for
libraries that must be linked to system emulator targets, $(LIBS_TOOLS)
is used for tools like qemu-img, qemu-nbd, etc and $(LIBS_QGA) is used
for the QEMU guest agent. There is currently no specific variable for
the userspace emulator targets as the global $(LIBS), or more targeted
variables shown below, are sufficient.

In addition to these variables, it is possible to provide cflags and
libs against individual source code files, by defining variables of the
form $FILENAME-cflags and $FILENAME-libs. For example, the curl block
driver needs to link to the libcurl library, so block/Makefile defines
some variables:

  curl.o-cflags      := $(CURL_CFLAGS)
  curl.o-libs        := $(CURL_LIBS)

The scope is a little different between the two variables. The libs get
used when linking any target binary that includes the curl.o object
file, while the cflags get used when compiling the curl.c file only.


Statically defined files
------------------------

The following key files are statically defined in the source tree, with
the rules needed to build QEMU. Their behaviour is influenced by a
number of dynamically created files listed later.

- Makefile

The main entry point used when invoking make to build all the components
of QEMU. The default 'all' target will naturally result in the build of
every component. The various tools and helper binaries are built
directly via a non-recursive set of rules.

Each system/userspace emulation target needs to have a slightly
different set of make rules / variables. Thus, make will be recursively
invoked for each of the emulation targets.

The recursive invocation will end up processing the toplevel
Makefile.target file (more on that later).


- */Makefile.objs

Since the source code is spread across multiple directories, the rules
for each file are similarly modularized. Thus each subdirectory
containing .c files will usually also contain a Makefile.objs file.
These files are not directly invoked by a recursive make, but instead
they are imported by the top level Makefile and/or Makefile.target

Each Makefile.objs usually just declares a set of variables listing the
.o files that need building from the source files in the directory. They
will also define any custom linker or compiler flags. For example in
block/Makefile.objs

  block-obj-$(CONFIG_LIBISCSI) += iscsi.o
  block-obj-$(CONFIG_CURL) += curl.o

  ..snip...

  iscsi.o-cflags     := $(LIBISCSI_CFLAGS)
  iscsi.o-libs       := $(LIBISCSI_LIBS)
  curl.o-cflags      := $(CURL_CFLAGS)
  curl.o-libs        := $(CURL_LIBS)

If there are any rules defined in the Makefile.objs file, they should
all use $(obj) as a prefix to the target, e.g.

  $(obj)/generated-tcg-tracers.h: $(obj)/generated-tcg-tracers.h-timestamp


- Makefile.target

This file provides the entry point used to build each individual system
or userspace emulator target. Each enabled target has its own
subdirectory. For example if configure is run with the argument
'--target-list=x86_64-softmmu', then a sub-directory 'x86_64-softmu'
will be created, containing a 'Makefile' which symlinks back to
Makefile.target

So when the recursive '$(MAKE) -C x86_64-softmmu' is invoked, it ends up
using Makefile.target for the build rules.


- rules.mak

This file provides the generic helper rules for invoking build tools, in
particular the compiler and linker. This also contains the magic (hairy)
'unnest-vars' function which is used to merge the variable definitions
from all Makefile.objs in the source tree down into the main Makefile
context.


- default-configs/*.mak

The files under default-configs/ control what emulated hardware is built
into each QEMU system and userspace emulator targets. They merely
contain a long list of config variable definitions. For example,
default-configs/x86_64-softmmu.mak has:

  include pci.mak
  include sound.mak
  include usb.mak
  CONFIG_QXL=$(CONFIG_SPICE)
  CONFIG_VGA_ISA=y
  CONFIG_VGA_CIRRUS=y
  CONFIG_VMWARE_VGA=y
  CONFIG_VIRTIO_VGA=y
  ...snip...

These files rarely need changing unless new devices / hardware need to
be enabled for a particular system/userspace emulation target


- tests/Makefile

Rules for building the unit tests. This file is included directly by the
top level Makefile, so anything defined in this file will influence the
entire build system. Care needs to be taken when writing rules for tests
to ensure they only apply to the unit test execution / build.

- tests/docker/Makefile.include

Rules for Docker tests. Like tests/Makefile, this file is included
directly by the top level Makefile, anything defined in this file will
influence the entire build system.

- po/Makefile

Rules for building and installing the binary message catalogs from the
text .po file sources. This almost never needs changing for any reason.


Dynamically created files
-------------------------

The following files are generated dynamically by configure in order to
control the behaviour of the statically defined makefiles. This avoids
the need for QEMU makefiles to go through any pre-processing as seen
with autotools, where Makefile.am generates Makefile.in which generates
Makefile.


- config-host.mak

When configure has determined the characteristics of the build host it
will write a long list of variables to config-host.mak file. This
provides the various install directories, compiler / linker flags and a
variety of CONFIG_* variables related to optionally enabled features.
This is imported by the top level Makefile in order to tailor the build
output.

The variables defined here are those which are applicable to all QEMU
build outputs. Variables which are potentially different for each
emulator target are defined by the next file...

It is also used as a dependency checking mechanism. If make sees that
the modification timestamp on configure is newer than that on
config-host.mak, then configure will be re-run.


- config-host.h

The config-host.h file is used by source code to determine what features
are enabled. It is generated from the contents of config-host.mak using
the scripts/create_config program. This extracts all the CONFIG_* variables,
most of the HOST_* variables and a few other misc variables from
config-host.mak, formatting them as C preprocessor macros.


- $TARGET-NAME/config-target.mak

TARGET-NAME is the name of a system or userspace emulator, for example,
x86_64-softmmu denotes the system emulator for the x86_64 architecture.
This file contains the variables which need to vary on a per-target
basis. For example, it will indicate whether KVM or Xen are enabled for
the target and any other potential custom libraries needed for linking
the target.


- $TARGET-NAME/config-devices.mak

TARGET-NAME is again the name of a system or userspace emulator. The
config-devices.mak file is automatically generated by make using the
scripts/make_device_config.sh program, feeding it the
default-configs/$TARGET-NAME file as input.


- $TARGET-NAME/Makefile

This is the entrypoint used when make recurses to build a single system
or userspace emulator target. It is merely a symlink back to the
Makefile.target in the top level.


Useful make targets
===================

- help

  Print a help message for the most common build targets.

- print-VAR

  Print the value of the variable VAR. Useful for debugging the build
  system.