| ===================== |
| VFIO device Migration |
| ===================== |
| |
| Migration of virtual machine involves saving the state for each device that |
| the guest is running on source host and restoring this saved state on the |
| destination host. This document details how saving and restoring of VFIO |
| devices is done in QEMU. |
| |
| Migration of VFIO devices consists of two phases: the optional pre-copy phase, |
| and the stop-and-copy phase. The pre-copy phase is iterative and allows to |
| accommodate VFIO devices that have a large amount of data that needs to be |
| transferred. The iterative pre-copy phase of migration allows for the guest to |
| continue whilst the VFIO device state is transferred to the destination, this |
| helps to reduce the total downtime of the VM. VFIO devices can choose to skip |
| the pre-copy phase of migration by returning pending_bytes as zero during the |
| pre-copy phase. |
| |
| A detailed description of the UAPI for VFIO device migration can be found in |
| the comment for the ``vfio_device_migration_info`` structure in the header |
| file linux-headers/linux/vfio.h. |
| |
| VFIO implements the device hooks for the iterative approach as follows: |
| |
| * A ``save_setup`` function that sets up the migration region and sets _SAVING |
| flag in the VFIO device state. |
| |
| * A ``load_setup`` function that sets up the migration region on the |
| destination and sets _RESUMING flag in the VFIO device state. |
| |
| * A ``save_live_pending`` function that reads pending_bytes from the vendor |
| driver, which indicates the amount of data that the vendor driver has yet to |
| save for the VFIO device. |
| |
| * A ``save_live_iterate`` function that reads the VFIO device's data from the |
| vendor driver through the migration region during iterative phase. |
| |
| * A ``save_state`` function to save the device config space if it is present. |
| |
| * A ``save_live_complete_precopy`` function that resets _RUNNING flag from the |
| VFIO device state and iteratively copies the remaining data for the VFIO |
| device until the vendor driver indicates that no data remains (pending bytes |
| is zero). |
| |
| * A ``load_state`` function that loads the config section and the data |
| sections that are generated by the save functions above |
| |
| * ``cleanup`` functions for both save and load that perform any migration |
| related cleanup, including unmapping the migration region |
| |
| |
| The VFIO migration code uses a VM state change handler to change the VFIO |
| device state when the VM state changes from running to not-running, and |
| vice versa. |
| |
| Similarly, a migration state change handler is used to trigger a transition of |
| the VFIO device state when certain changes of the migration state occur. For |
| example, the VFIO device state is transitioned back to _RUNNING in case a |
| migration failed or was canceled. |
| |
| System memory dirty pages tracking |
| ---------------------------------- |
| |
| A ``log_global_start`` and ``log_global_stop`` memory listener callback informs |
| the VFIO IOMMU module to start and stop dirty page tracking. A ``log_sync`` |
| memory listener callback marks those system memory pages as dirty which are |
| used for DMA by the VFIO device. The dirty pages bitmap is queried per |
| container. All pages pinned by the vendor driver through external APIs have to |
| be marked as dirty during migration. When there are CPU writes, CPU dirty page |
| tracking can identify dirtied pages, but any page pinned by the vendor driver |
| can also be written by the device. There is currently no device or IOMMU |
| support for dirty page tracking in hardware. |
| |
| By default, dirty pages are tracked when the device is in pre-copy as well as |
| stop-and-copy phase. So, a page pinned by the vendor driver will be copied to |
| the destination in both phases. Copying dirty pages in pre-copy phase helps |
| QEMU to predict if it can achieve its downtime tolerances. If QEMU during |
| pre-copy phase keeps finding dirty pages continuously, then it understands |
| that even in stop-and-copy phase, it is likely to find dirty pages and can |
| predict the downtime accordingly. |
| |
| QEMU also provides a per device opt-out option ``pre-copy-dirty-page-tracking`` |
| which disables querying the dirty bitmap during pre-copy phase. If it is set to |
| off, all dirty pages will be copied to the destination in stop-and-copy phase |
| only. |
| |
| System memory dirty pages tracking when vIOMMU is enabled |
| --------------------------------------------------------- |
| |
| With vIOMMU, an IO virtual address range can get unmapped while in pre-copy |
| phase of migration. In that case, the unmap ioctl returns any dirty pages in |
| that range and QEMU reports corresponding guest physical pages dirty. During |
| stop-and-copy phase, an IOMMU notifier is used to get a callback for mapped |
| pages and then dirty pages bitmap is fetched from VFIO IOMMU modules for those |
| mapped ranges. |
| |
| Flow of state changes during Live migration |
| =========================================== |
| |
| Below is the flow of state change during live migration. |
| The values in the brackets represent the VM state, the migration state, and |
| the VFIO device state, respectively. |
| |
| Live migration save path |
| ------------------------ |
| |
| :: |
| |
| QEMU normal running state |
| (RUNNING, _NONE, _RUNNING) |
| | |
| migrate_init spawns migration_thread |
| Migration thread then calls each device's .save_setup() |
| (RUNNING, _SETUP, _RUNNING|_SAVING) |
| | |
| (RUNNING, _ACTIVE, _RUNNING|_SAVING) |
| If device is active, get pending_bytes by .save_live_pending() |
| If total pending_bytes >= threshold_size, call .save_live_iterate() |
| Data of VFIO device for pre-copy phase is copied |
| Iterate till total pending bytes converge and are less than threshold |
| | |
| On migration completion, vCPU stops and calls .save_live_complete_precopy for |
| each active device. The VFIO device is then transitioned into _SAVING state |
| (FINISH_MIGRATE, _DEVICE, _SAVING) |
| | |
| For the VFIO device, iterate in .save_live_complete_precopy until |
| pending data is 0 |
| (FINISH_MIGRATE, _DEVICE, _STOPPED) |
| | |
| (FINISH_MIGRATE, _COMPLETED, _STOPPED) |
| Migraton thread schedules cleanup bottom half and exits |
| |
| Live migration resume path |
| -------------------------- |
| |
| :: |
| |
| Incoming migration calls .load_setup for each device |
| (RESTORE_VM, _ACTIVE, _STOPPED) |
| | |
| For each device, .load_state is called for that device section data |
| (RESTORE_VM, _ACTIVE, _RESUMING) |
| | |
| At the end, .load_cleanup is called for each device and vCPUs are started |
| (RUNNING, _NONE, _RUNNING) |
| |
| Postcopy |
| ======== |
| |
| Postcopy migration is currently not supported for VFIO devices. |
