docs/markdown/Simple-comparison.md - meson - Git at Google

 # A simple comparison

 In this experiment we generated one thousand C files with contents
 that looked like this.

 ```c
 #include<stdio.h>
 #include"header.h"

 int func23() { return 0; }
 ```

 The function number was different in each file. In addition there was
 a main C file that just called each function in turn. We then
 generated build system files for *Meson*, *CMake*, *SCons*, *Premake*
 and *Autotools* that compiled these files into a single executable.

 With this we measured three different things. The first is
 configuration time, that is, the time the build system takes to
 generate necessary build files. This is usually called the *configure
 step*. The time was measured in seconds.

 The second thing to measure was the build time. This should be limited
 by the compiler and in the optimal case should be the same for every
 build system. Four parallel processes were used in this test.

 The third thing we measured was the empty build time. This measures
 how much time the build system takes to check the states of all source
 files because if any of them could potentially cause a rebuild.

 Since CMake has two different backends, Make and Ninja, we ran the
 tests on both of them. All tests were run on a 2011 era MacBook Pro
 running Ubuntu 13/04. The tests were run multiple times and we always
 took the fastest time.

 Here are the results for configuration time.

 ![Configuration times](images/conftime.png)

 The reason SCons got zero seconds on this test is because you cannot
 separate configure and build steps. They run as one unit. Autotools is
 the clear loser of this test as it is over an order of magnitude
 slower than the second slowest one. This configuration time includes
 both autogen and configure. All other systems take less than one
 second to do this setup, which is fast enough for a human being to
 interpret as instantaneous.

 ![Build times](https://raw.githubusercontent.com/wiki/jpakkane/meson/buildtime.png)

 Build times are a bit more even. SCons is the slowest, being almost
 ten seconds slower than the second slowest. Some of it is work from
 the configure step but it still has the worst performance. Premake is
 the fastest Make-based build system narrowly beating out Autotools.
 Both Ninja-based build systems are faster than all non-Ninja ones with
 Meson being slightly faster. In practice the difference is minimal.
 The advantages of Ninja can be seen by comparing CMake's times when
 using Make or Ninja. It is possible to shave off 3.5 seconds (over
 20%) of the total build time just by changing the backend. The
 project's CMake configuration files don't need any changes.

 ![No-op time](https://raw.githubusercontent.com/wiki/jpakkane/meson/emptytime.png)

 Empty build times reflect the performance of regular build times.
 SCons is again the slowest taking over three seconds compared to
 Meson, which takes only 0.03 seconds, a difference of two orders of
 magnitude. Even Autotools, the fastest Make-based system, is almost
 one order of magnitude slower. Ninja holds the top spots just like in
 the previous test.

 Conclusions
 -----

 Build system performance matters. Even with this extremely simple
 example we can find differences between various popular build systems.
 As the project size increases, these differences grow even larger.
 (The author has witnessed no-op build times of 30 seconds for Make
 versus less than one second for Ninja when compiling the Clang
 compiler.) Keeping incremental build times low is one of the major
 keys of programmer productivity as it allows developers to iterate
 faster and stay in the creative zone.

 Original scripts
 -----

 Those who want to run these experiments themselves can download the scripts here:

 * [Generator script](https://raw.githubusercontent.com/wiki/jpakkane/meson/gen_src.py)
 * [Measurement script](https://raw.githubusercontent.com/wiki/jpakkane/meson/measure.py)
	# A simple comparison

	In this experiment we generated one thousand C files with contents
	that looked like this.

	```c
	#include<stdio.h>
	#include"header.h"

	int func23() { return 0; }
	```

	The function number was different in each file. In addition there was
	a main C file that just called each function in turn. We then
	generated build system files for Meson, CMake, SCons, Premake
	and Autotools that compiled these files into a single executable.

	With this we measured three different things. The first is
	configuration time, that is, the time the build system takes to
	generate necessary build files. This is usually called the *configure
	step*. The time was measured in seconds.

	The second thing to measure was the build time. This should be limited
	by the compiler and in the optimal case should be the same for every
	build system. Four parallel processes were used in this test.

	The third thing we measured was the empty build time. This measures
	how much time the build system takes to check the states of all source
	files because if any of them could potentially cause a rebuild.

	Since CMake has two different backends, Make and Ninja, we ran the
	tests on both of them. All tests were run on a 2011 era MacBook Pro
	running Ubuntu 13/04. The tests were run multiple times and we always
	took the fastest time.

	Here are the results for configuration time.

	![Configuration times](images/conftime.png)

	The reason SCons got zero seconds on this test is because you cannot
	separate configure and build steps. They run as one unit. Autotools is
	the clear loser of this test as it is over an order of magnitude
	slower than the second slowest one. This configuration time includes
	both autogen and configure. All other systems take less than one
	second to do this setup, which is fast enough for a human being to
	interpret as instantaneous.

	![Build times](https://raw.githubusercontent.com/wiki/jpakkane/meson/buildtime.png)

	Build times are a bit more even. SCons is the slowest, being almost
	ten seconds slower than the second slowest. Some of it is work from
	the configure step but it still has the worst performance. Premake is
	the fastest Make-based build system narrowly beating out Autotools.
	Both Ninja-based build systems are faster than all non-Ninja ones with
	Meson being slightly faster. In practice the difference is minimal.
	The advantages of Ninja can be seen by comparing CMake's times when
	using Make or Ninja. It is possible to shave off 3.5 seconds (over
	20%) of the total build time just by changing the backend. The
	project's CMake configuration files don't need any changes.

	![No-op time](https://raw.githubusercontent.com/wiki/jpakkane/meson/emptytime.png)

	Empty build times reflect the performance of regular build times.
	SCons is again the slowest taking over three seconds compared to
	Meson, which takes only 0.03 seconds, a difference of two orders of
	magnitude. Even Autotools, the fastest Make-based system, is almost
	one order of magnitude slower. Ninja holds the top spots just like in
	the previous test.

	Conclusions
	-----

	Build system performance matters. Even with this extremely simple
	example we can find differences between various popular build systems.
	As the project size increases, these differences grow even larger.
	(The author has witnessed no-op build times of 30 seconds for Make
	versus less than one second for Ninja when compiling the Clang
	compiler.) Keeping incremental build times low is one of the major
	keys of programmer productivity as it allows developers to iterate
	faster and stay in the creative zone.

	Original scripts
	-----

	Those who want to run these experiments themselves can download the scripts here:

	* [Generator script](https://raw.githubusercontent.com/wiki/jpakkane/meson/gen_src.py)
	* [Measurement script](https://raw.githubusercontent.com/wiki/jpakkane/meson/measure.py)