| Block replication |
| ---------------------------------------- |
| Copyright Fujitsu, Corp. 2016 |
| Copyright (c) 2016 Intel Corporation |
| Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. |
| |
| This work is licensed under the terms of the GNU GPL, version 2 or later. |
| See the COPYING file in the top-level directory. |
| |
| Block replication is used for continuous checkpoints. It is designed |
| for COLO (COarse-grain LOck-stepping) where the Secondary VM is running. |
| It can also be applied for FT/HA (Fault-tolerance/High Assurance) scenario, |
| where the Secondary VM is not running. |
| |
| This document gives an overview of block replication's design. |
| |
| == Background == |
| High availability solutions such as micro checkpoint and COLO will do |
| consecutive checkpoints. The VM state of the Primary and Secondary VM is |
| identical right after a VM checkpoint, but becomes different as the VM |
| executes till the next checkpoint. To support disk contents checkpoint, |
| the modified disk contents in the Secondary VM must be buffered, and are |
| only dropped at next checkpoint time. To reduce the network transportation |
| effort during a vmstate checkpoint, the disk modification operations of |
| the Primary disk are asynchronously forwarded to the Secondary node. |
| |
| == Workflow == |
| The following is the image of block replication workflow: |
| |
| +----------------------+ +------------------------+ |
| |Primary Write Requests| |Secondary Write Requests| |
| +----------------------+ +------------------------+ |
| | | |
| | (4) |
| | V |
| | /-------------\ |
| | Copy and Forward | | |
| |---------(1)----------+ | Disk Buffer | |
| | | | | |
| | (3) \-------------/ |
| | speculative ^ |
| | write through (2) |
| | | | |
| V V | |
| +--------------+ +----------------+ |
| | Primary Disk | | Secondary Disk | |
| +--------------+ +----------------+ |
| |
| 1) Primary write requests will be copied and forwarded to Secondary |
| QEMU. |
| 2) Before Primary write requests are written to Secondary disk, the |
| original sector content will be read from Secondary disk and |
| buffered in the Disk buffer, but it will not overwrite the existing |
| sector content (it could be from either "Secondary Write Requests" or |
| previous COW of "Primary Write Requests") in the Disk buffer. |
| 3) Primary write requests will be written to Secondary disk. |
| 4) Secondary write requests will be buffered in the Disk buffer and it |
| will overwrite the existing sector content in the buffer. |
| |
| == Architecture == |
| We are going to implement block replication from many basic |
| blocks that are already in QEMU. |
| |
| virtio-blk || |
| ^ || .---------- |
| | || | Secondary |
| 1 Quorum || '---------- |
| / \ || virtio-blk |
| / \ || ^ |
| Primary 2 filter | |
| disk ^ 7 Quorum |
| | / |
| 3 NBD -------> 3 NBD / |
| client || server 2 filter |
| || ^ ^ |
| --------. || | | |
| Primary | || Secondary disk <--------- hidden-disk 5 <--------- active-disk 4 |
| --------' || | backing ^ backing |
| || | | |
| || | | |
| || '-------------------------' |
| || blockdev-backup sync=none 6 |
| |
| 1) The disk on the primary is represented by a block device with two |
| children, providing replication between a primary disk and the host that |
| runs the secondary VM. The read pattern (fifo) for quorum can be extended |
| to make the primary always read from the local disk instead of going through |
| NBD. |
| |
| 2) The new block filter (the name is replication) will control the block |
| replication. |
| |
| 3) The secondary disk receives writes from the primary VM through QEMU's |
| embedded NBD server (speculative write-through). |
| |
| 4) The disk on the secondary is represented by a custom block device |
| (called active-disk). It should start as an empty disk, and the format |
| should support bdrv_make_empty() and backing file. |
| |
| 5) The hidden-disk is created automatically. It buffers the original content |
| that is modified by the primary VM. It should also start as an empty disk, |
| and the driver supports bdrv_make_empty() and backing file. |
| |
| 6) The blockdev-backup job (sync=none) is run to allow hidden-disk to buffer |
| any state that would otherwise be lost by the speculative write-through |
| of the NBD server into the secondary disk. So before block replication, |
| the primary disk and secondary disk should contain the same data. |
| |
| 7) The secondary also has a quorum node, so after secondary failover it |
| can become the new primary and continue replication. |
| |
| |
| == Failure Handling == |
| There are 7 internal errors when block replication is running: |
| 1. I/O error on primary disk |
| 2. Forwarding primary write requests failed |
| 3. Backup failed |
| 4. I/O error on secondary disk |
| 5. I/O error on active disk |
| 6. Making active disk or hidden disk empty failed |
| 7. Doing failover failed |
| In case 1 and 5, we just report the error to the disk layer. In case 2, 3, |
| 4 and 6, we just report block replication's error to FT/HA manager (which |
| decides when to do a new checkpoint, when to do failover). |
| In case 7, if active commit failed, we use replication failover failed state |
| in Secondary's write operation (what decides which target to write). |
| |
| == New block driver interface == |
| We add four block driver interfaces to control block replication: |
| a. replication_start_all() |
| Start block replication, called in migration/checkpoint thread. |
| We must call block_replication_start_all() in secondary QEMU before |
| calling block_replication_start_all() in primary QEMU. The caller |
| must hold the I/O mutex lock if it is in migration/checkpoint |
| thread. |
| b. replication_do_checkpoint_all() |
| This interface is called after all VM state is transferred to |
| Secondary QEMU. The Disk buffer will be dropped in this interface. |
| The caller must hold the I/O mutex lock if it is in migration/checkpoint |
| thread. |
| c. replication_get_error_all() |
| This interface is called to check if error happened in replication. |
| The caller must hold the I/O mutex lock if it is in migration/checkpoint |
| thread. |
| d. replication_stop_all() |
| It is called on failover. We will flush the Disk buffer into |
| Secondary Disk and stop block replication. The vm should be stopped |
| before calling it if you use this API to shutdown the guest, or other |
| things except failover. The caller must hold the I/O mutex lock if it is |
| in migration/checkpoint thread. |
| |
| == Usage == |
| Primary: |
| -drive if=xxx,driver=quorum,read-pattern=fifo,id=colo1,vote-threshold=1,\ |
| children.0.file.filename=1.raw,\ |
| children.0.driver=raw |
| |
| Run qmp command in primary qemu: |
| { "execute": "human-monitor-command", |
| "arguments": { |
| "command-line": "drive_add -n buddy driver=replication,mode=primary,file.driver=nbd,file.host=xxxx,file.port=xxxx,file.export=colo1,node-name=nbd_client1" |
| } |
| } |
| { "execute": "x-blockdev-change", |
| "arguments": { |
| "parent": "colo1", |
| "node": "nbd_client1" |
| } |
| } |
| Note: |
| 1. There should be only one NBD Client for each primary disk. |
| 2. host is the secondary physical machine's hostname or IP |
| 3. Each disk must have its own export name. |
| 4. It is all a single argument to -drive and you should ignore the |
| leading whitespace. |
| 5. The qmp command line must be run after running qmp command line in |
| secondary qemu. |
| 6. After primary failover we need remove children.1 (replication driver). |
| |
| Secondary: |
| -drive if=none,driver=raw,file.filename=1.raw,id=colo1 \ |
| -drive if=none,id=childs1,driver=replication,mode=secondary,top-id=top-disk1 |
| file.file.filename=active_disk.qcow2,\ |
| file.driver=qcow2,\ |
| file.backing.file.filename=hidden_disk.qcow2,\ |
| file.backing.driver=qcow2,\ |
| file.backing.backing=colo1 |
| -drive if=xxx,driver=quorum,read-pattern=fifo,id=top-disk1,\ |
| vote-threshold=1,children.0=childs1 |
| |
| Then run qmp command in secondary qemu: |
| { "execute": "nbd-server-start", |
| "arguments": { |
| "addr": { |
| "type": "inet", |
| "data": { |
| "host": "xxx", |
| "port": "xxx" |
| } |
| } |
| } |
| } |
| { "execute": "nbd-server-add", |
| "arguments": { |
| "device": "colo1", |
| "writable": true |
| } |
| } |
| |
| Note: |
| 1. The export name in secondary QEMU command line is the secondary |
| disk's id. |
| 2. The export name for the same disk must be the same |
| 3. The qmp command nbd-server-start and nbd-server-add must be run |
| before running the qmp command migrate on primary QEMU |
| 4. Active disk, hidden disk and nbd target's length should be the |
| same. |
| 5. It is better to put active disk and hidden disk in ramdisk. |
| 6. It is all a single argument to -drive, and you should ignore |
| the leading whitespace. |
| |
| After Failover: |
| Primary: |
| The secondary host is down, so we should run the following qmp command |
| to remove the nbd child from the quorum: |
| { "execute": "x-blockdev-change", |
| "arguments": { |
| "parent": "colo1", |
| "child": "children.1" |
| } |
| } |
| { "execute": "human-monitor-command", |
| "arguments": { |
| "command-line": "drive_del xxxx" |
| } |
| } |
| Note: there is no qmp command to remove the blockdev now |
| |
| Secondary: |
| The primary host is down, so we should do the following thing: |
| { "execute": "nbd-server-stop" } |
| |
| Promote Secondary to Primary: |
| see COLO-FT.txt |
| |
| TODO: |
| 1. Shared disk |
