SPDK has support for a virtual NVMe controller called nvmf/vfio-user. These are instructions on how to use it with the QEMU vfio-user client.
You will need QEMU 10.1.1 or later. Let's build it:
cd ~/src/ curl -L https://download.qemu.org/qemu-10.1.1.tar.xz | tar xJf - cd ~/src/qemu-10.1.1 ./configure --enable-kvm --enable-vnc --target-list=x86_64-softmmu --enable-trace-backends=log --enable-debug make -j
Here we'll use SPDK v25.09:
git clone https://github.com/spdk/spdk --recursive --branch v25.09 spdk-v25.09 cd spdk-v25.09 ./configure --with-vfio-user make -j
NB: SPDK includes libvfio-user as a submodule: some older versions of SPDK require a particular branch of libvfio-user.
./build/bin/nvmf_tgt --no-huge -s 1024 &
Now let's create an NVMe controller with a 512MB RAM-based namespace:
mkdir /tmp/spdk scripts/rpc.py bdev_malloc_create 512 512 -b Malloc0 scripts/rpc.py nvmf_create_subsystem nqn.2019-07.io.spdk:cnode0 -a -s SPDK0 scripts/rpc.py nvmf_subsystem_add_ns nqn.2019-07.io.spdk:cnode0 Malloc0 scripts/rpc.py nvmf_create_transport -t VFIOUSER scripts/rpc.py nvmf_subsystem_add_listener nqn.2019-07.io.spdk:cnode0 -t VFIOUSER -a /tmp/spdk -s 0
Now let‘s start our guest VM. We’ll create a 2GB VM booting from an Ubuntu cloud image, with a NIC so we can ssh in, and our virtual NVMe PCI device:
cd ~/src/qemu-10.1.1
./build/qemu-system-x86_64 \
-monitor stdio \
-machine accel=kvm,type=q35 \
-m 2G \
-object memory-backend-file,id=mem,size=2G,mem-path=/dev/shm/qemu-mem,share=on \
-numa node,memdev=mem \
-drive if=virtio,format=qcow2,file=/home/jlevon/bionic-server-cloudimg-amd64.img \
-device virtio-net-pci,netdev=net0 \
-netdev user,id=net0,hostfwd=tcp::2222-:22
-device '{"driver":"vfio-user-pci","socket":{"path": "/tmp/spdk/cntrl", "type": "unix"}}'
And in the VM we should be able to see our NVMe device:
ssh -p 2222 ubuntu@localhost ubuntu@cloudimg:~$ sudo nvme list Node SN Model Namespace Usage Format FW Rev ---------------- -------------------- ---------------------------------------- --------- -------------------------- ---------------- -------- /dev/nvme0n1 SPDK0 SPDK bdev Controller 1 536.87 MB / 536.87 MB 512 B + 0 B 25.05
For generating a cloud image, see below.
To use the nvmf/vfio-user target with a libvirt quest, in addition to the libvirtd configuration documented in the README the guest RAM must be backed by hugepages:
<memoryBacking>
<hugepages>
<page size='2048' unit='KiB'/>
</hugepages>
<source type='memfd'/>
<access mode='shared'/>
</memoryBacking>
Because SPDK must be run as root, either fix the vfio-user socket permissions or configure libvirt to run QEMU as root.
Live migration with SPDK is currently non-functional, although code exists in libvfio-user. If you are interested in helping, please let us know!
Set up the necessary metadata files:
sudo apt install cloud-image-utils $ cat metadata.yaml instance-id: iid-local01 local-hostname: cloudimg $ cat user-data.yaml #cloud-config ssh_import_id: - gh:jlevon cloud-localds seed.img user-data.yaml metadata.yaml
Replace jlevon with your github username; this will pull your public key from github, so you can subsequently ssh into the VM.