docs/qcow2-cache.txt - qemu - Git at Google

 qcow2 L2/refcount cache configuration
 =====================================
 Copyright (C) 2015, 2018-2020 Igalia, S.L.
 Author: Alberto Garcia <berto@igalia.com>

 This work is licensed under the terms of the GNU GPL, version 2 or
 later. See the COPYING file in the top-level directory.

 Introduction
 ------------
 The QEMU qcow2 driver has two caches that can improve the I/O
 performance significantly. However, setting the right cache sizes is
 not a straightforward operation.

 This document attempts to give an overview of the L2 and refcount
 caches, and how to configure them.

 Please refer to the docs/interop/qcow2.txt file for an in-depth
 technical description of the qcow2 file format.


 Clusters
 --------
 A qcow2 file is organized in units of constant size called clusters.

 The cluster size is configurable, but it must be a power of two and
 its value 512 bytes or higher. QEMU currently defaults to 64 KB
 clusters, and it does not support sizes larger than 2MB.

 The 'qemu-img create' command supports specifying the size using the
 cluster_size option:

    qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G


 The L2 tables
 -------------
 The qcow2 format uses a two-level structure to map the virtual disk as
 seen by the guest to the disk image in the host. These structures are
 called the L1 and L2 tables.

 There is one single L1 table per disk image. The table is small and is
 always kept in memory.

 There can be many L2 tables, depending on how much space has been
 allocated in the image. Each table is one cluster in size. In order to
 read or write data from the virtual disk, QEMU needs to read its
 corresponding L2 table to find out where that data is located. Since
 reading the table for each I/O operation can be expensive, QEMU keeps
 an L2 cache in memory to speed up disk access.

 The size of the L2 cache can be configured, and setting the right
 value can improve the I/O performance significantly.


 The refcount blocks
 -------------------
 The qcow2 format also maintains a reference count for each cluster.
 Reference counts are used for cluster allocation and internal
 snapshots. The data is stored in a two-level structure similar to the
 L1/L2 tables described above.

 The second level structures are called refcount blocks, are also one
 cluster in size and the number is also variable and dependent on the
 amount of allocated space.

 Each block contains a number of refcount entries. Their size (in bits)
 is a power of two and must not be higher than 64. It defaults to 16
 bits, but a different value can be set using the refcount_bits option:

    qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G

 QEMU keeps a refcount cache to speed up I/O much like the
 aforementioned L2 cache, and its size can also be configured.


 Choosing the right cache sizes
 ------------------------------
 In order to choose the cache sizes we need to know how they relate to
 the amount of allocated space.

 The part of the virtual disk that can be mapped by the L2 and refcount
 caches (in bytes) is:

    disk_size = l2_cache_size * cluster_size / 8
    disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits

 With the default values for cluster_size (64KB) and refcount_bits
 (16), this becomes:

    disk_size = l2_cache_size * 8192
    disk_size = refcount_cache_size * 32768

 So in order to cover n GB of disk space with the default values we
 need:

    l2_cache_size = disk_size_GB * 131072
    refcount_cache_size = disk_size_GB * 32768

 For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual
 image size (given that the default cluster size is used):

    8 GB / 8192 = 1 MB

 The refcount cache is 4 times the cluster size by default. With the default
 cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for
 8 GB of image size:

    262144 * 32768 = 8 GB


 How to configure the cache sizes
 --------------------------------
 Cache sizes can be configured using the -drive option in the
 command-line, or the 'blockdev-add' QMP command.

 There are three options available, and all of them take bytes:

 "l2-cache-size":         maximum size of the L2 table cache
 "refcount-cache-size":   maximum size of the refcount block cache
 "cache-size":            maximum size of both caches combined

 There are a few things that need to be taken into account:

  - Both caches must have a size that is a multiple of the cluster size
    (or the cache entry size: see "Using smaller cache sizes" below).

  - The maximum L2 cache size is 32 MB by default on Linux platforms (enough
    for full coverage of 256 GB images, with the default cluster size). This
    value can be modified using the "l2-cache-size" option. QEMU will not use
    more memory than needed to hold all of the image's L2 tables, regardless
    of this max. value.
    On non-Linux platforms the maximal value is smaller by default (8 MB) and
    this difference stems from the fact that on Linux the cache can be cleared
    periodically if needed, using the "cache-clean-interval" option (see below).
    The minimal L2 cache size is 2 clusters (or 2 cache entries, see below).

  - The default (and minimum) refcount cache size is 4 clusters.

  - If only "cache-size" is specified then QEMU will assign as much
    memory as possible to the L2 cache before increasing the refcount
    cache size.

  - At most two of "l2-cache-size", "refcount-cache-size", and "cache-size"
    can be set simultaneously.

 Unlike L2 tables, refcount blocks are not used during normal I/O but
 only during allocations and internal snapshots. In most cases they are
 accessed sequentially (even during random guest I/O) so increasing the
 refcount cache size won't have any measurable effect in performance
 (this can change if you are using internal snapshots, so you may want
 to think about increasing the cache size if you use them heavily).

 Before QEMU 2.12 the refcount cache had a default size of 1/4 of the
 L2 cache size. This resulted in unnecessarily large caches, so now the
 refcount cache is as small as possible unless overridden by the user.


 Using smaller cache entries
 ---------------------------
 The qcow2 L2 cache can store complete tables. This means that if QEMU
 needs an entry from an L2 table then the whole table is read from disk
 and is kept in the cache. If the cache is full then a complete table
 needs to be evicted first.

 This can be inefficient with large cluster sizes since it results in
 more disk I/O and wastes more cache memory.

 Since QEMU 2.12 you can change the size of the L2 cache entry and make
 it smaller than the cluster size. This can be configured using the
 "l2-cache-entry-size" parameter:

    -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096

 Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or
 the cluster size if it's smaller).

 Some things to take into account:

  - The L2 cache entry size has the same restrictions as the cluster
    size (power of two, at least 512 bytes).

  - Smaller entry sizes generally improve the cache efficiency and make
    disk I/O faster. This is particularly true with solid state drives
    so it's a good idea to reduce the entry size in those cases. With
    rotating hard drives the situation is a bit more complicated so you
    should test it first and stay with the default size if unsure.

  - Try different entry sizes to see which one gives faster performance
    in your case. The block size of the host filesystem is generally a
    good default (usually 4096 bytes in the case of ext4, hence the
    default).

  - Only the L2 cache can be configured this way. The refcount cache
    always uses the cluster size as the entry size.

  - If the L2 cache is big enough to hold all of the image's L2 tables
    (as explained in the "Choosing the right cache sizes" and "How to
    configure the cache sizes" sections in this document) then none of
    this is necessary and you can omit the "l2-cache-entry-size"
    parameter altogether. In this case QEMU makes the entry size
    equal to the cluster size by default.


 Reducing the memory usage
 -------------------------
 It is possible to clean unused cache entries in order to reduce the
 memory usage during periods of low I/O activity.

 The parameter "cache-clean-interval" defines an interval (in seconds),
 after which all the cache entries that haven't been accessed during the
 interval are removed from memory. Setting this parameter to 0 disables this
 feature.

 The following example removes all unused cache entries every 15 minutes:

    -drive file=hd.qcow2,cache-clean-interval=900

 If unset, the default value for this parameter is 600 on platforms which
 support this functionality, and is 0 (disabled) on other platforms.

 This functionality currently relies on the MADV_DONTNEED argument for
 madvise() to actually free the memory. This is a Linux-specific feature,
 so cache-clean-interval is not supported on other systems.


 Extended L2 Entries
 -------------------
 All numbers shown in this document are valid for qcow2 images with normal
 64-bit L2 entries.

 Images with extended L2 entries need twice as much L2 metadata, so the L2
 cache size must be twice as large for the same disk space.

    disk_size = l2_cache_size * cluster_size / 16

 i.e.

    l2_cache_size = disk_size * 16 / cluster_size

 Refcount blocks are not affected by this.
	qcow2 L2/refcount cache configuration
	=====================================
	Copyright (C) 2015, 2018-2020 Igalia, S.L.
	Author: Alberto Garcia <berto@igalia.com>

	This work is licensed under the terms of the GNU GPL, version 2 or
	later. See the COPYING file in the top-level directory.

	Introduction
	------------
	The QEMU qcow2 driver has two caches that can improve the I/O
	performance significantly. However, setting the right cache sizes is
	not a straightforward operation.

	This document attempts to give an overview of the L2 and refcount
	caches, and how to configure them.

	Please refer to the docs/interop/qcow2.txt file for an in-depth
	technical description of the qcow2 file format.


	Clusters
	--------
	A qcow2 file is organized in units of constant size called clusters.

	The cluster size is configurable, but it must be a power of two and
	its value 512 bytes or higher. QEMU currently defaults to 64 KB
	clusters, and it does not support sizes larger than 2MB.

	The 'qemu-img create' command supports specifying the size using the
	cluster_size option:

	qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G


	The L2 tables
	-------------
	The qcow2 format uses a two-level structure to map the virtual disk as
	seen by the guest to the disk image in the host. These structures are
	called the L1 and L2 tables.

	There is one single L1 table per disk image. The table is small and is
	always kept in memory.

	There can be many L2 tables, depending on how much space has been
	allocated in the image. Each table is one cluster in size. In order to
	read or write data from the virtual disk, QEMU needs to read its
	corresponding L2 table to find out where that data is located. Since
	reading the table for each I/O operation can be expensive, QEMU keeps
	an L2 cache in memory to speed up disk access.

	The size of the L2 cache can be configured, and setting the right
	value can improve the I/O performance significantly.


	The refcount blocks
	-------------------
	The qcow2 format also maintains a reference count for each cluster.
	Reference counts are used for cluster allocation and internal
	snapshots. The data is stored in a two-level structure similar to the
	L1/L2 tables described above.

	The second level structures are called refcount blocks, are also one
	cluster in size and the number is also variable and dependent on the
	amount of allocated space.

	Each block contains a number of refcount entries. Their size (in bits)
	is a power of two and must not be higher than 64. It defaults to 16
	bits, but a different value can be set using the refcount_bits option:

	qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G

	QEMU keeps a refcount cache to speed up I/O much like the
	aforementioned L2 cache, and its size can also be configured.


	Choosing the right cache sizes
	------------------------------
	In order to choose the cache sizes we need to know how they relate to
	the amount of allocated space.

	The part of the virtual disk that can be mapped by the L2 and refcount
	caches (in bytes) is:

	disk_size = l2_cache_size * cluster_size / 8
	disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits

	With the default values for cluster_size (64KB) and refcount_bits
	(16), this becomes:

	disk_size = l2_cache_size * 8192
	disk_size = refcount_cache_size * 32768

	So in order to cover n GB of disk space with the default values we
	need:

	l2_cache_size = disk_size_GB * 131072
	refcount_cache_size = disk_size_GB * 32768

	For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual
	image size (given that the default cluster size is used):

	8 GB / 8192 = 1 MB

	The refcount cache is 4 times the cluster size by default. With the default
	cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for
	8 GB of image size:

	262144 * 32768 = 8 GB


	How to configure the cache sizes
	--------------------------------
	Cache sizes can be configured using the -drive option in the
	command-line, or the 'blockdev-add' QMP command.

	There are three options available, and all of them take bytes:

	"l2-cache-size": maximum size of the L2 table cache
	"refcount-cache-size": maximum size of the refcount block cache
	"cache-size": maximum size of both caches combined

	There are a few things that need to be taken into account:

	- Both caches must have a size that is a multiple of the cluster size
	(or the cache entry size: see "Using smaller cache sizes" below).

	- The maximum L2 cache size is 32 MB by default on Linux platforms (enough
	for full coverage of 256 GB images, with the default cluster size). This
	value can be modified using the "l2-cache-size" option. QEMU will not use
	more memory than needed to hold all of the image's L2 tables, regardless
	of this max. value.
	On non-Linux platforms the maximal value is smaller by default (8 MB) and
	this difference stems from the fact that on Linux the cache can be cleared
	periodically if needed, using the "cache-clean-interval" option (see below).
	The minimal L2 cache size is 2 clusters (or 2 cache entries, see below).

	- The default (and minimum) refcount cache size is 4 clusters.

	- If only "cache-size" is specified then QEMU will assign as much
	memory as possible to the L2 cache before increasing the refcount
	cache size.

	- At most two of "l2-cache-size", "refcount-cache-size", and "cache-size"
	can be set simultaneously.

	Unlike L2 tables, refcount blocks are not used during normal I/O but
	only during allocations and internal snapshots. In most cases they are
	accessed sequentially (even during random guest I/O) so increasing the
	refcount cache size won't have any measurable effect in performance
	(this can change if you are using internal snapshots, so you may want
	to think about increasing the cache size if you use them heavily).

	Before QEMU 2.12 the refcount cache had a default size of 1/4 of the
	L2 cache size. This resulted in unnecessarily large caches, so now the
	refcount cache is as small as possible unless overridden by the user.


	Using smaller cache entries
	---------------------------
	The qcow2 L2 cache can store complete tables. This means that if QEMU
	needs an entry from an L2 table then the whole table is read from disk
	and is kept in the cache. If the cache is full then a complete table
	needs to be evicted first.

	This can be inefficient with large cluster sizes since it results in
	more disk I/O and wastes more cache memory.

	Since QEMU 2.12 you can change the size of the L2 cache entry and make
	it smaller than the cluster size. This can be configured using the
	"l2-cache-entry-size" parameter:

	-drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096

	Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or
	the cluster size if it's smaller).

	Some things to take into account:

	- The L2 cache entry size has the same restrictions as the cluster
	size (power of two, at least 512 bytes).

	- Smaller entry sizes generally improve the cache efficiency and make
	disk I/O faster. This is particularly true with solid state drives
	so it's a good idea to reduce the entry size in those cases. With
	rotating hard drives the situation is a bit more complicated so you
	should test it first and stay with the default size if unsure.

	- Try different entry sizes to see which one gives faster performance
	in your case. The block size of the host filesystem is generally a
	good default (usually 4096 bytes in the case of ext4, hence the
	default).

	- Only the L2 cache can be configured this way. The refcount cache
	always uses the cluster size as the entry size.

	- If the L2 cache is big enough to hold all of the image's L2 tables
	(as explained in the "Choosing the right cache sizes" and "How to
	configure the cache sizes" sections in this document) then none of
	this is necessary and you can omit the "l2-cache-entry-size"
	parameter altogether. In this case QEMU makes the entry size
	equal to the cluster size by default.


	Reducing the memory usage
	-------------------------
	It is possible to clean unused cache entries in order to reduce the
	memory usage during periods of low I/O activity.

	The parameter "cache-clean-interval" defines an interval (in seconds),
	after which all the cache entries that haven't been accessed during the
	interval are removed from memory. Setting this parameter to 0 disables this
	feature.

	The following example removes all unused cache entries every 15 minutes:

	-drive file=hd.qcow2,cache-clean-interval=900

	If unset, the default value for this parameter is 600 on platforms which
	support this functionality, and is 0 (disabled) on other platforms.

	This functionality currently relies on the MADV_DONTNEED argument for
	madvise() to actually free the memory. This is a Linux-specific feature,
	so cache-clean-interval is not supported on other systems.


	Extended L2 Entries
	-------------------
	All numbers shown in this document are valid for qcow2 images with normal
	64-bit L2 entries.

	Images with extended L2 entries need twice as much L2 metadata, so the L2
	cache size must be twice as large for the same disk space.

	disk_size = l2_cache_size * cluster_size / 16

	i.e.

	l2_cache_size = disk_size * 16 / cluster_size

	Refcount blocks are not affected by this.