SPDK and libvfio-user

SPDK v21.01 added experimental support for a virtual NVMe controller called nvmf/vfio-user. The controller can be used with the same QEMU command line as the one used for GPIO.

Build QEMU

Use Oracle's QEMU vfio-user-p3.1 from https://github.com/oracle/qemu:

git clone https://github.com/oracle/qemu qemu-orcl --branch vfio-user-p3.1
cd qemu-orcl
git submodule update --init --recursive
./configure --enable-multiprocess
make

Build SPDK

Use SPDK v23.05:

git clone https://github.com/spdk/spdk --branch v23.05
cd spdk
git submodule update --init --recursive
./configure --with-vfio-user
make

Start SPDK:

LD_LIBRARY_PATH=build/lib:dpdk/build/lib build/bin/nvmf_tgt &

Create an NVMe controller with a 512MB RAM-based namespace:

rm -f /var/run/{cntrl,bar0}
scripts/rpc.py nvmf_create_transport -t VFIOUSER && \
	scripts/rpc.py bdev_malloc_create 512 512 -b Malloc0 && \
	scripts/rpc.py nvmf_create_subsystem nqn.2019-07.io.spdk:cnode0 -a -s SPDK0 && \
	scripts/rpc.py nvmf_subsystem_add_ns nqn.2019-07.io.spdk:cnode0 Malloc0 && \
	scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode0 -t VFIOUSER -a /var/run -s 0

Start the Guest

Start the guest with e.g. 4 GB of RAM:

qemu-orcl/build/qemu-system-x86_64 \
	-m 4G -object memory-backend-file,id=mem0,size=4G,mem-path=/dev/hugepages,share=on,prealloc=yes -numa node,memdev=mem0 \
	-device vfio-user-pci,socket=/var/run/cntrl

Live Migration

SPDK v22.01 has experimental support for live migration. This CR contains additional fixes that make live migration more reliable. Check it out and build SPDK as explained in Build SPDK, both on the source and on the destination hosts.

Then build QEMU as explained in Build QEMU using the following version:

https://github.com/oracle/qemu/tree/vfio-user-dbfix

Start the guest at the source host as explained in Start the Guest, appending the x-enable-migration=on argument to the vfio-user-pci option.

Then, at the destination host, start the nvmf/vfio-user target and QEMU, passing the -incoming option to QEMU:

-incoming tcp:0:4444

QEMU will block at the destination waiting for the guest to be migrated.

Bear in mind that if the guest‘s disk don’t reside in shared storage you‘ll get I/O errors soon after migration. The easiest way around this is to put the guest’s disk on some NFS mount and share between the source and destination hosts.

Finally, migrate the guest by issuing the migrate command on the QEMU monitor (enter CTRL-A + C to enter the monitor):

migrate -d tcp:<destination host IP address>:4444

Migration should happen almost instantaneously, there's no message to show that migration finished neither in the source nor on the destination hosts. Simply hitting ENTER at the destination is enough to tell that migration finished.

Finally, type q in the source QEMU monitor to exit source QEMU.

For more information in live migration see https://www.linux-kvm.org/page/Migration.

Note that the above live migration code in qemu and SPDK relies on the older live migration format, this is kept in the https://github.com/nutanix/libvfio-user/tree/migration-v1 branch of libvfio-user.

libvirt

To use the nvmf/vfio-user target with a libvirt quest, in addition to the libvirtd configuration documented in the README the guest RAM must be backed by hugepages:

<memoryBacking>
  <hugepages>
    <page size='2048' unit='KiB'/>
  </hugepages>
  <source type='memfd'/>
  <access mode='shared'/>
</memoryBacking>

Because SPDK must be run as root, either fix the vfio-user socket permissions or configure libvirt to run QEMU as root.