vdpa net
============

This document explains the setup and usage of the vdpa network device.
The vdpa network device is a paravirtualized vdpa emulate device.

Description
-----------

VDPA net devices support dirty page bitmap mark and vring state saving and recovery.

Users can use this VDPA device for live migration simulation testing in a nested virtualization environment.

Registers layout
----------------

The vdpa device add live migrate registers layout as follow::

  Offset       Register Name	        Bitwidth     Associated vq
  0x0          LM_LOGGING_CTRL          4bits
  0x10         LM_BASE_ADDR_LOW         32bits
  0x14         LM_BASE_ADDR_HIGH        32bits
  0x18         LM_END_ADDR_LOW          32bits
  0x1c         LM_END_ADDR_HIGH         32bits
  0x20         LM_RING_STATE_OFFSET	32bits       vq0
  0x24         LM_RING_STATE_OFFSET	32bits       vq1
  0x28         LM_RING_STATE_OFFSET	32bits       vq2
  ......
  0x20+1023*4  LM_RING_STATE_OFFSET     32bits       vq1023

These registers are extended at the end of the notify bar space.

Architecture diagram
--------------------
::

  |------------------------------------------------------------------------|
  | guest-L1-user-space                                                    |
  |                                                                        |
  |                               |----------------------------------------|
  |                               |       [virtio-net driver]              |
  |                               |              ^  guest-L2-src(iommu=on) |
  |                               |--------------|-------------------------|
  |                               |              |  qemu-L2-src(viommu)    |
  | [dpdk-vdpa]<->[vhost socket]<-+->[vhost-user backend(iommu=on)]        |
  --------------------------------------------------------------------------
  --------------------------------------------------------------------------
  |       ^                             guest-L1-kernel-space              |
  |       |                                                                |
  |    [VFIO]                                                              |
  |       ^                                                                |
  |       |                             guest-L1-src(iommu=on)             |
  --------|-----------------------------------------------------------------
  --------|-----------------------------------------------------------------
  | [vdpa net device(iommu=on)]        [manager nic device]                |
  |          |                                    |                        |
  |          |                                    |                        |
  |     [tap device]     qemu-L1-src(viommu)      |                        |
  ------------------------------------------------+-------------------------
                                                  |
                                                  |
                        ---------------------     |
                        | kernel net bridge |<-----
                        |     virbr0        |<----------------------------------
                        ---------------------                                  |
                                                                               |
                                                                               |
  --------------------------------------------------------------------------   |
  | guest-L1-user-space                                                    |   |
  |                                                                        |   |
  |                               |----------------------------------------|   |
  |                               |       [virtio-net driver]              |   |
  |                               |              ^  guest-L2-dst(iommu=on) |   |
  |                               |--------------|-------------------------|   |
  |                               |              |  qemu-L2-dst(viommu)    |   |
  | [dpdk-vdpa]<->[vhost socket]<-+->[vhost-user backend(iommu=on)]        |   |
  --------------------------------------------------------------------------   |
  --------------------------------------------------------------------------   |
  |       ^                             guest-L1-kernel-space              |   |
  |       |                                                                |   |
  |    [VFIO]                                                              |   |
  |       ^                                                                |   |
  |       |                             guest-L1-dst(iommu=on)             |   |
  --------|-----------------------------------------------------------------   |
  --------|-----------------------------------------------------------------   |
  | [vdpa net device(iommu=on)]        [manager nic device]----------------+----
  |          |                                                             |
  |          |                                                             |
  |     [tap device]     qemu-L1-dst(viommu)                               |
  --------------------------------------------------------------------------


Device properties
-----------------

The Virtio vdpa device can be configured with the following properties:

 * ``vdpa=on`` open vdpa device emulated.

Usages
--------
This patch add virtio sriov support and vdpa live migrate support.
You can open vdpa by set xml file as follow::

  <qemu:commandline  xmlns:qemu='http://libvirt.org/schemas/domain/qemu/1.0'>
  <qemu:arg value='-device'/>
  <qemu:arg value='intel-iommu,intremap=on,device-iotlb=on,aw-bits=48'/>
  <qemu:arg value='-netdev'/>
  <qemu:arg value='tap,id=hostnet1,script=no,downscript=no,vhost=off'/>
  <qemu:arg value='-device'/>
  <qemu:arg value='virtio-net-pci,netdev=hostnet1,id=net1,mac=56:4a:b7:4f:4d:a9,bus=pci.6,addr=0x0,iommu_platform=on,ats=on,vdpa=on'/>
  </qemu:commandline>

Limitations
-----------
1. Dependent on tap device with param ``vhost=off``.
2. Nested virtualization environment only supports ``q35`` machines.
3. Current only support split vring live migrate.