# A simple comparison

In this experiment we generated one thousand C files with contents that looked like this.

```c
#include<stdio.h>
#include"header.h"

int func23() { return 0; }
```

The function number was different in each file. In addition there was a main C file that just called each function in turn. We then generated build system files for *Meson*, *CMake*, *SCons*, *Premake* and *Autotools* that compiled these files into a single executable.

With this we measured three different things. The first is configuration time, that is, the time the build system takes to generate necessary build files. This is usually called the *configure step*. The time was measured in seconds.

The second thing to measure was the build time. This should be limited by the compiler and in the optimal case should be the same for every build system. Four parallel processes were used in this test.

The third thing we measured was the empty build time. This measures how much time the build system takes to check the states of all source files because if any of them could potentially cause a rebuild.

Since CMake has two different backends, Make and Ninja, we ran the tests on both of them. All tests were run on a 2011 era Macbook Pro running Ubuntu 13/04. The tests were run multiple times and we always took the fastest time.

Here are the results for configuration time.

![Configuration times](images/conftime.png)

The reason SCons got zero seconds on this test is because you cannot separate configure and build steps. They run as one unit. Autotools is the clear loser of this test as it is over an order of magnitude slower than the second slowest one. This configuration time includes both autogen and configure. All other systems take less than one second to do this setup, which is fast enough for a human being to interpret as instantaneous.

![Build times](https://raw.githubusercontent.com/wiki/jpakkane/meson/buildtime.png)

Build times are a bit more even. SCons is the slowest, being almost ten seconds slower than the second slowest. Some of it is work from the configure step but it still has the worst performance. Premake is the fastest Make-based build system narrowly beating out Autotools. Both Ninja-based build systems are faster than all non-Ninja ones with Meson being slightly faster. In practice the difference is minimal. The advantages of Ninja can be seen by comparing CMake's times when using Make or Ninja. It is possible to shave off 3.5 seconds (over 20%) of the total build time just by changing the backend. The project's CMake configuration files don't need any changes.

![No-op time](https://raw.githubusercontent.com/wiki/jpakkane/meson/emptytime.png)

Empty build times reflect the performance of regular build times. SCons is again the slowest taking over three seconds compared to Meson, which takes only 0.03 seconds, a difference of two orders of magnitude. Even Autotools, the fastest Make-based system, is almost one order of magnitude slower. Ninja holds the top spots just like in the previous test.

Conclusions
-----

Build system performance matters. Even with this extremely simple example we can find differences between various popular build systems. As the project size increases, these differences grow even larger. (The author has witnessed no-op build times of 30 seconds for Make versus less than one second for Ninja when compiling the Clang compiler.) Keeping incremental build times low is one of the major keys of programmer productivity as it allows developers to iterate faster and stay in the creative zone.

Original scripts
-----

Those who want to run these experiments themselves can download the scripts here:

* [Generator script](https://raw.githubusercontent.com/wiki/jpakkane/meson/gen_src.py)
* [Measurement script](https://raw.githubusercontent.com/wiki/jpakkane/meson/measure.py)