| COLO-proxy | |
| ---------- | |
| Copyright (c) 2016 Intel Corporation | |
| Copyright (c) 2016 HUAWEI TECHNOLOGIES CO., LTD. | |
| Copyright (c) 2016 Fujitsu, Corp. | |
| This work is licensed under the terms of the GNU GPL, version 2 or later. | |
| See the COPYING file in the top-level directory. | |
| This document gives an overview of COLO proxy's design. | |
| == Background == | |
| COLO-proxy is a part of COLO project. It is used | |
| to compare the network package to help COLO decide | |
| whether to do checkpoint. With COLO-proxy's help, | |
| COLO greatly improves the performance. | |
| The filter-redirector, filter-mirror, colo-compare | |
| and filter-rewriter compose the COLO-proxy. | |
| == Architecture == | |
| COLO-Proxy is based on qemu netfilter and it's a plugin for qemu netfilter | |
| (except colo-compare). It keep Secondary VM connect normally to | |
| client and compare packets sent by PVM with sent by SVM. | |
| If the packet difference, notify COLO-frame to do checkpoint and send | |
| all primary packet has queued. Otherwise just send the queued primary | |
| packet and drop the queued secondary packet. | |
| Below is a COLO proxy ascii figure: | |
| Primary qemu Secondary qemu | |
| +--------------------------------------------------------------+ +----------------------------------------------------------------+ | |
| | +----------------------------------------------------------+ | | +-----------------------------------------------------------+ | | |
| | | | | | | | | | |
| | | guest | | | | guest | | | |
| | | | | | | | | | |
| | +-------^--------------------------+-----------------------+ | | +---------------------+--------+----------------------------+ | | |
| | | | | | ^ | | | |
| | | | | | | | | | |
| | | +------------------------------------------------------+ | | | | | |
| |netfilter| | | | | | netfilter | | | | |
| | +----------+ +----------------------------+ | | | +-----------------------------------------------------------+ | | |
| | | | | | | out | | | | | | filter execute order | | | |
| | | | | +-----------------------------+ | | | | | | +-------------------> | | | |
| | | | | | | | | | | | | | | TCP | | | |
| | | +-----+--+-+ +-----v----+ +-----v----+ |pri +----+----+sec| | | | +------------+ +---+----+---v+rewriter++ +------------+ | | | |
| | | | | | | | | |in | |in | | | | | | | | | | | | | | |
| | | | filter | | filter | | filter +------> colo <------+ +--------> filter +--> adjust | adjust +--> filter | | | | |
| | | | mirror | |redirector| |redirector| | | compare | | | | | | redirector | | ack | seq | | redirector | | | | |
| | | | | | | | | | | | | | | | | | | | | | | | | | |
| | | +----^-----+ +----+-----+ +----------+ | +---------+ | | | | +------------+ +--------+--------------+ +---+--------+ | | | |
| | | | tx | rx rx | | | | | tx all | rx | | | |
| | | | | | | | | +-----------------------------------------------------------+ | | |
| | | | +--------------+ | | | | | | | |
| | | | filter execute order | | | | | | | | |
| | | | +----------------> | | | +--------------------------------------------------------+ | | |
| | +-----------------------------------------+ | | | | |
| | | | | | | | |
| +--------------------------------------------------------------+ +----------------------------------------------------------------+ | |
| |guest receive | guest send | |
| | | | |
| +--------+----------------------------v------------------------+ | |
| | | NOTE: filter direction is rx/tx/all | |
| | tap | rx:receive packets sent to the netdev | |
| | | tx:receive packets sent by the netdev | |
| +--------------------------------------------------------------+ | |
| 1.Guest receive packet route: | |
| Primary: | |
| Tap --> Mirror Client Filter | |
| Mirror client will send packet to guest,at the | |
| same time, copy and forward packet to secondary | |
| mirror server. | |
| Secondary: | |
| Mirror Server Filter --> TCP Rewriter | |
| If receive packet is TCP packet,we will adjust ack | |
| and update TCP checksum, then send to secondary | |
| guest. Otherwise directly send to guest. | |
| 2.Guest send packet route: | |
| Primary: | |
| Guest --> Redirect Server Filter | |
| Redirect server filter receive primary guest packet | |
| but do nothing, just pass to next filter. | |
| Redirect Server Filter --> COLO-Compare | |
| COLO-compare receive primary guest packet then | |
| waiting secondary redirect packet to compare it. | |
| If packet same,send queued primary packet and clear | |
| queued secondary packet, Otherwise send primary packet | |
| and do checkpoint. | |
| COLO-Compare --> Another Redirector Filter | |
| The redirector get packet from colo-compare by use | |
| chardev socket. | |
| Redirector Filter --> Tap | |
| Send the packet. | |
| Secondary: | |
| Guest --> TCP Rewriter Filter | |
| If the packet is TCP packet,we will adjust seq | |
| and update TCP checksum. Then send it to | |
| redirect client filter. Otherwise directly send to | |
| redirect client filter. | |
| Redirect Client Filter --> Redirect Server Filter | |
| Forward packet to primary. | |
| == Components introduction == | |
| Filter-mirror is a netfilter plugin. | |
| It gives qemu the ability to mirror | |
| packets to a chardev. | |
| Filter-redirector is a netfilter plugin. | |
| It gives qemu the ability to redirect net packet. | |
| Redirector can redirect filter's net packet to outdev, | |
| and redirect indev's packet to filter. | |
| filter | |
| + | |
| redirector | | |
| +--------------+ | |
| | | | | |
| | | | | |
| | | | | |
| indev +---------+ +----------> outdev | |
| | | | | |
| | | | | |
| | | | | |
| +--------------+ | |
| | | |
| v | |
| filter | |
| COLO-compare, we do packet comparing job. | |
| Packets coming from the primary char indev will be sent to outdev. | |
| Packets coming from the secondary char dev will be dropped after comparing. | |
| COLO-compare needs two input chardevs and one output chardev: | |
| primary_in=chardev1-id (source: primary send packet) | |
| secondary_in=chardev2-id (source: secondary send packet) | |
| outdev=chardev3-id | |
| Filter-rewriter will rewrite some of secondary packet to make | |
| secondary guest's tcp connection established successfully. | |
| In this module we will rewrite tcp packet's ack to the secondary | |
| from primary,and rewrite tcp packet's seq to the primary from | |
| secondary. | |
| == Usage == | |
| Here is an example using demonstration IP and port addresses to more | |
| clearly describe the usage. | |
| Primary(ip:3.3.3.3): | |
| -netdev tap,id=hn0,vhost=off | |
| -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 | |
| -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off | |
| -chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off | |
| -chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off | |
| -chardev socket,id=compare0-0,host=3.3.3.3,port=9001 | |
| -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off | |
| -chardev socket,id=compare_out0,host=3.3.3.3,port=9005 | |
| -object iothread,id=iothread1 | |
| -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0 | |
| -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out | |
| -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0 | |
| -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,iothread=iothread1 | |
| Secondary(ip:3.3.3.8): | |
| -netdev tap,id=hn0,vhost=off | |
| -device e1000,netdev=hn0,mac=52:a4:00:12:78:66 | |
| -chardev socket,id=red0,host=3.3.3.3,port=9003 | |
| -chardev socket,id=red1,host=3.3.3.3,port=9004 | |
| -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0 | |
| -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1 | |
| -object filter-rewriter,id=f3,netdev=hn0,queue=all | |
| If you want to use virtio-net-pci or other driver with vnet_header: | |
| Primary(ip:3.3.3.3): | |
| -netdev tap,id=hn0,vhost=off,script=/etc/qemu-ifup,downscript=/etc/qemu-ifdown | |
| -device e1000,id=e0,netdev=hn0,mac=52:a4:00:12:78:66 | |
| -chardev socket,id=mirror0,host=3.3.3.3,port=9003,server=on,wait=off | |
| -chardev socket,id=compare1,host=3.3.3.3,port=9004,server=on,wait=off | |
| -chardev socket,id=compare0,host=3.3.3.3,port=9001,server=on,wait=off | |
| -chardev socket,id=compare0-0,host=3.3.3.3,port=9001 | |
| -chardev socket,id=compare_out,host=3.3.3.3,port=9005,server=on,wait=off | |
| -chardev socket,id=compare_out0,host=3.3.3.3,port=9005 | |
| -object filter-mirror,id=m0,netdev=hn0,queue=tx,outdev=mirror0,vnet_hdr_support | |
| -object filter-redirector,netdev=hn0,id=redire0,queue=rx,indev=compare_out,vnet_hdr_support | |
| -object filter-redirector,netdev=hn0,id=redire1,queue=rx,outdev=compare0,vnet_hdr_support | |
| -object colo-compare,id=comp0,primary_in=compare0-0,secondary_in=compare1,outdev=compare_out0,vnet_hdr_support | |
| Secondary(ip:3.3.3.8): | |
| -netdev tap,id=hn0,vhost=off | |
| -device e1000,netdev=hn0,mac=52:a4:00:12:78:66 | |
| -chardev socket,id=red0,host=3.3.3.3,port=9003 | |
| -chardev socket,id=red1,host=3.3.3.3,port=9004 | |
| -object filter-redirector,id=f1,netdev=hn0,queue=tx,indev=red0,vnet_hdr_support | |
| -object filter-redirector,id=f2,netdev=hn0,queue=rx,outdev=red1,vnet_hdr_support | |
| -object filter-rewriter,id=f3,netdev=hn0,queue=all,vnet_hdr_support | |
| Note: | |
| a.COLO-proxy must work with COLO-frame and Block-replication. | |
| b.Primary COLO must be started firstly, because COLO-proxy needs | |
| chardev socket server running before secondary started. | |
| c.Filter-rewriter only rewrite tcp packet. |