| qcow2 L2/refcount cache configuration | |
| ===================================== | |
| Copyright (C) 2015, 2018-2020 Igalia, S.L. | |
| Author: Alberto Garcia <berto@igalia.com> | |
| This work is licensed under the terms of the GNU GPL, version 2 or | |
| later. See the COPYING file in the top-level directory. | |
| Introduction | |
| ------------ | |
| The QEMU qcow2 driver has two caches that can improve the I/O | |
| performance significantly. However, setting the right cache sizes is | |
| not a straightforward operation. | |
| This document attempts to give an overview of the L2 and refcount | |
| caches, and how to configure them. | |
| Please refer to the docs/interop/qcow2.rst file for an in-depth | |
| technical description of the qcow2 file format. | |
| Clusters | |
| -------- | |
| A qcow2 file is organized in units of constant size called clusters. | |
| The cluster size is configurable, but it must be a power of two and | |
| its value 512 bytes or higher. QEMU currently defaults to 64 KB | |
| clusters, and it does not support sizes larger than 2MB. | |
| The 'qemu-img create' command supports specifying the size using the | |
| cluster_size option: | |
| qemu-img create -f qcow2 -o cluster_size=128K hd.qcow2 4G | |
| The L2 tables | |
| ------------- | |
| The qcow2 format uses a two-level structure to map the virtual disk as | |
| seen by the guest to the disk image in the host. These structures are | |
| called the L1 and L2 tables. | |
| There is one single L1 table per disk image. The table is small and is | |
| always kept in memory. | |
| There can be many L2 tables, depending on how much space has been | |
| allocated in the image. Each table is one cluster in size. In order to | |
| read or write data from the virtual disk, QEMU needs to read its | |
| corresponding L2 table to find out where that data is located. Since | |
| reading the table for each I/O operation can be expensive, QEMU keeps | |
| an L2 cache in memory to speed up disk access. | |
| The size of the L2 cache can be configured, and setting the right | |
| value can improve the I/O performance significantly. | |
| The refcount blocks | |
| ------------------- | |
| The qcow2 format also maintains a reference count for each cluster. | |
| Reference counts are used for cluster allocation and internal | |
| snapshots. The data is stored in a two-level structure similar to the | |
| L1/L2 tables described above. | |
| The second level structures are called refcount blocks, are also one | |
| cluster in size and the number is also variable and dependent on the | |
| amount of allocated space. | |
| Each block contains a number of refcount entries. Their size (in bits) | |
| is a power of two and must not be higher than 64. It defaults to 16 | |
| bits, but a different value can be set using the refcount_bits option: | |
| qemu-img create -f qcow2 -o refcount_bits=8 hd.qcow2 4G | |
| QEMU keeps a refcount cache to speed up I/O much like the | |
| aforementioned L2 cache, and its size can also be configured. | |
| Choosing the right cache sizes | |
| ------------------------------ | |
| In order to choose the cache sizes we need to know how they relate to | |
| the amount of allocated space. | |
| The part of the virtual disk that can be mapped by the L2 and refcount | |
| caches (in bytes) is: | |
| disk_size = l2_cache_size * cluster_size / 8 | |
| disk_size = refcount_cache_size * cluster_size * 8 / refcount_bits | |
| With the default values for cluster_size (64KB) and refcount_bits | |
| (16), this becomes: | |
| disk_size = l2_cache_size * 8192 | |
| disk_size = refcount_cache_size * 32768 | |
| So in order to cover n GB of disk space with the default values we | |
| need: | |
| l2_cache_size = disk_size_GB * 131072 | |
| refcount_cache_size = disk_size_GB * 32768 | |
| For example, 1MB of L2 cache is needed to cover every 8 GB of the virtual | |
| image size (given that the default cluster size is used): | |
| 8 GB / 8192 = 1 MB | |
| The refcount cache is 4 times the cluster size by default. With the default | |
| cluster size of 64 KB, it is 256 KB (262144 bytes). This is sufficient for | |
| 8 GB of image size: | |
| 262144 * 32768 = 8 GB | |
| How to configure the cache sizes | |
| -------------------------------- | |
| Cache sizes can be configured using the -drive option in the | |
| command-line, or the 'blockdev-add' QMP command. | |
| There are three options available, and all of them take bytes: | |
| "l2-cache-size": maximum size of the L2 table cache | |
| "refcount-cache-size": maximum size of the refcount block cache | |
| "cache-size": maximum size of both caches combined | |
| There are a few things that need to be taken into account: | |
| - Both caches must have a size that is a multiple of the cluster size | |
| (or the cache entry size: see "Using smaller cache sizes" below). | |
| - The maximum L2 cache size is 32 MB by default on Linux platforms (enough | |
| for full coverage of 256 GB images, with the default cluster size). This | |
| value can be modified using the "l2-cache-size" option. QEMU will not use | |
| more memory than needed to hold all of the image's L2 tables, regardless | |
| of this max. value. | |
| On non-Linux platforms the maximal value is smaller by default (8 MB) and | |
| this difference stems from the fact that on Linux the cache can be cleared | |
| periodically if needed, using the "cache-clean-interval" option (see below). | |
| The minimal L2 cache size is 2 clusters (or 2 cache entries, see below). | |
| - The default (and minimum) refcount cache size is 4 clusters. | |
| - If only "cache-size" is specified then QEMU will assign as much | |
| memory as possible to the L2 cache before increasing the refcount | |
| cache size. | |
| - At most two of "l2-cache-size", "refcount-cache-size", and "cache-size" | |
| can be set simultaneously. | |
| Unlike L2 tables, refcount blocks are not used during normal I/O but | |
| only during allocations and internal snapshots. In most cases they are | |
| accessed sequentially (even during random guest I/O) so increasing the | |
| refcount cache size won't have any measurable effect in performance | |
| (this can change if you are using internal snapshots, so you may want | |
| to think about increasing the cache size if you use them heavily). | |
| Before QEMU 2.12 the refcount cache had a default size of 1/4 of the | |
| L2 cache size. This resulted in unnecessarily large caches, so now the | |
| refcount cache is as small as possible unless overridden by the user. | |
| Using smaller cache entries | |
| --------------------------- | |
| The qcow2 L2 cache can store complete tables. This means that if QEMU | |
| needs an entry from an L2 table then the whole table is read from disk | |
| and is kept in the cache. If the cache is full then a complete table | |
| needs to be evicted first. | |
| This can be inefficient with large cluster sizes since it results in | |
| more disk I/O and wastes more cache memory. | |
| Since QEMU 2.12 you can change the size of the L2 cache entry and make | |
| it smaller than the cluster size. This can be configured using the | |
| "l2-cache-entry-size" parameter: | |
| -drive file=hd.qcow2,l2-cache-size=2097152,l2-cache-entry-size=4096 | |
| Since QEMU 4.0 the value of l2-cache-entry-size defaults to 4KB (or | |
| the cluster size if it's smaller). | |
| Some things to take into account: | |
| - The L2 cache entry size has the same restrictions as the cluster | |
| size (power of two, at least 512 bytes). | |
| - Smaller entry sizes generally improve the cache efficiency and make | |
| disk I/O faster. This is particularly true with solid state drives | |
| so it's a good idea to reduce the entry size in those cases. With | |
| rotating hard drives the situation is a bit more complicated so you | |
| should test it first and stay with the default size if unsure. | |
| - Try different entry sizes to see which one gives faster performance | |
| in your case. The block size of the host filesystem is generally a | |
| good default (usually 4096 bytes in the case of ext4, hence the | |
| default). | |
| - Only the L2 cache can be configured this way. The refcount cache | |
| always uses the cluster size as the entry size. | |
| - If the L2 cache is big enough to hold all of the image's L2 tables | |
| (as explained in the "Choosing the right cache sizes" and "How to | |
| configure the cache sizes" sections in this document) then none of | |
| this is necessary and you can omit the "l2-cache-entry-size" | |
| parameter altogether. In this case QEMU makes the entry size | |
| equal to the cluster size by default. | |
| Reducing the memory usage | |
| ------------------------- | |
| It is possible to clean unused cache entries in order to reduce the | |
| memory usage during periods of low I/O activity. | |
| The parameter "cache-clean-interval" defines an interval (in seconds), | |
| after which all the cache entries that haven't been accessed during the | |
| interval are removed from memory. Setting this parameter to 0 disables this | |
| feature. | |
| The following example removes all unused cache entries every 15 minutes: | |
| -drive file=hd.qcow2,cache-clean-interval=900 | |
| If unset, the default value for this parameter is 600 on platforms which | |
| support this functionality, and is 0 (disabled) on other platforms. | |
| This functionality currently relies on the MADV_DONTNEED argument for | |
| madvise() to actually free the memory. This is a Linux-specific feature, | |
| so cache-clean-interval is not supported on other systems. | |
| Extended L2 Entries | |
| ------------------- | |
| All numbers shown in this document are valid for qcow2 images with normal | |
| 64-bit L2 entries. | |
| Images with extended L2 entries need twice as much L2 metadata, so the L2 | |
| cache size must be twice as large for the same disk space. | |
| disk_size = l2_cache_size * cluster_size / 16 | |
| i.e. | |
| l2_cache_size = disk_size * 16 / cluster_size | |
| Refcount blocks are not affected by this. |