Vhost-user Protocol
===================

Copyright (c) 2014 Virtual Open Systems Sarl.

This work is licensed under the terms of the GNU GPL, version 2 or later.
See the COPYING file in the top-level directory.
===================

This protocol is aiming to complement the ioctl interface used to control the
vhost implementation in the Linux kernel. It implements the control plane needed
to establish virtqueue sharing with a user space process on the same host. It
uses communication over a Unix domain socket to share file descriptors in the
ancillary data of the message.

The protocol defines 2 sides of the communication, master and slave. Master is
the application that shares its virtqueues, in our case QEMU. Slave is the
consumer of the virtqueues.

In the current implementation QEMU is the Master, and the Slave is intended to
be a software Ethernet switch running in user space, such as Snabbswitch.

Master and slave can be either a client (i.e. connecting) or server (listening)
in the socket communication.

Message Specification
---------------------

Note that all numbers are in the machine native byte order. A vhost-user message
consists of 3 header fields and a payload:

------------------------------------
| request | flags | size | payload |
------------------------------------

 * Request: 32-bit type of the request
 * Flags: 32-bit bit field:
   - Lower 2 bits are the version (currently 0x01)
   - Bit 2 is the reply flag - needs to be sent on each reply from the slave
 * Size - 32-bit size of the payload


Depending on the request type, payload can be:

 * A single 64-bit integer
   -------
   | u64 |
   -------

   u64: a 64-bit unsigned integer

 * A vring state description
   ---------------
  | index | num |
  ---------------

   Index: a 32-bit index
   Num: a 32-bit number

 * A vring address description
   --------------------------------------------------------------
   | index | flags | size | descriptor | used | available | log |
   --------------------------------------------------------------

   Index: a 32-bit vring index
   Flags: a 32-bit vring flags
   Descriptor: a 64-bit user address of the vring descriptor table
   Used: a 64-bit user address of the vring used ring
   Available: a 64-bit user address of the vring available ring
   Log: a 64-bit guest address for logging

 * Memory regions description
   ---------------------------------------------------
   | num regions | padding | region0 | ... | region7 |
   ---------------------------------------------------

   Num regions: a 32-bit number of regions
   Padding: 32-bit

   A region is:
   -----------------------------------------------------
   | guest address | size | user address | mmap offset |
   -----------------------------------------------------

   Guest address: a 64-bit guest address of the region
   Size: a 64-bit size
   User address: a 64-bit user address
   mmap offset: 64-bit offset where region starts in the mapped memory

In QEMU the vhost-user message is implemented with the following struct:

typedef struct VhostUserMsg {
    VhostUserRequest request;
    uint32_t flags;
    uint32_t size;
    union {
        uint64_t u64;
        struct vhost_vring_state state;
        struct vhost_vring_addr addr;
        VhostUserMemory memory;
    };
} QEMU_PACKED VhostUserMsg;

Communication
-------------

The protocol for vhost-user is based on the existing implementation of vhost
for the Linux Kernel. Most messages that can be sent via the Unix domain socket
implementing vhost-user have an equivalent ioctl to the kernel implementation.

The communication consists of master sending message requests and slave sending
message replies. Most of the requests don't require replies. Here is a list of
the ones that do:

 * VHOST_GET_FEATURES
 * VHOST_GET_VRING_BASE

There are several messages that the master sends with file descriptors passed
in the ancillary data:

 * VHOST_SET_MEM_TABLE
 * VHOST_SET_LOG_FD
 * VHOST_SET_VRING_KICK
 * VHOST_SET_VRING_CALL
 * VHOST_SET_VRING_ERR

If Master is unable to send the full message or receives a wrong reply it will
close the connection. An optional reconnection mechanism can be implemented.

Multi queue support
-------------------
The protocol supports multiple queues by setting all index fields in the sent
messages to a properly calculated value.

Message types
-------------

 * VHOST_USER_GET_FEATURES

      Id: 1
      Equivalent ioctl: VHOST_GET_FEATURES
      Master payload: N/A
      Slave payload: u64

      Get from the underlying vhost implementation the features bitmask.

 * VHOST_USER_SET_FEATURES

      Id: 2
      Ioctl: VHOST_SET_FEATURES
      Master payload: u64

      Enable features in the underlying vhost implementation using a bitmask.

 * VHOST_USER_SET_OWNER

      Id: 3
      Equivalent ioctl: VHOST_SET_OWNER
      Master payload: N/A

      Issued when a new connection is established. It sets the current Master
      as an owner of the session. This can be used on the Slave as a
      "session start" flag.

 * VHOST_USER_RESET_OWNER

      Id: 4
      Equivalent ioctl: VHOST_RESET_OWNER
      Master payload: N/A

      Issued when a new connection is about to be closed. The Master will no
      longer own this connection (and will usually close it).

 * VHOST_USER_SET_MEM_TABLE

      Id: 5
      Equivalent ioctl: VHOST_SET_MEM_TABLE
      Master payload: memory regions description

      Sets the memory map regions on the slave so it can translate the vring
      addresses. In the ancillary data there is an array of file descriptors
      for each memory mapped region. The size and ordering of the fds matches
      the number and ordering of memory regions.

 * VHOST_USER_SET_LOG_BASE

      Id: 6
      Equivalent ioctl: VHOST_SET_LOG_BASE
      Master payload: u64

      Sets the logging base address.

 * VHOST_USER_SET_LOG_FD

      Id: 7
      Equivalent ioctl: VHOST_SET_LOG_FD
      Master payload: N/A

      Sets the logging file descriptor, which is passed as ancillary data.

 * VHOST_USER_SET_VRING_NUM

      Id: 8
      Equivalent ioctl: VHOST_SET_VRING_NUM
      Master payload: vring state description

      Sets the number of vrings for this owner.

 * VHOST_USER_SET_VRING_ADDR

      Id: 9
      Equivalent ioctl: VHOST_SET_VRING_ADDR
      Master payload: vring address description
      Slave payload: N/A

      Sets the addresses of the different aspects of the vring.

 * VHOST_USER_SET_VRING_BASE

      Id: 10
      Equivalent ioctl: VHOST_SET_VRING_BASE
      Master payload: vring state description

      Sets the base offset in the available vring.

 * VHOST_USER_GET_VRING_BASE

      Id: 11
      Equivalent ioctl: VHOST_USER_GET_VRING_BASE
      Master payload: vring state description
      Slave payload: vring state description

      Get the available vring base offset.

 * VHOST_USER_SET_VRING_KICK

      Id: 12
      Equivalent ioctl: VHOST_SET_VRING_KICK
      Master payload: u64

      Set the event file descriptor for adding buffers to the vring. It
      is passed in the ancillary data.
      Bits (0-7) of the payload contain the vring index. Bit 8 is the
      invalid FD flag. This flag is set when there is no file descriptor
      in the ancillary data. This signals that polling should be used
      instead of waiting for a kick.

 * VHOST_USER_SET_VRING_CALL

      Id: 13
      Equivalent ioctl: VHOST_SET_VRING_CALL
      Master payload: u64

      Set the event file descriptor to signal when buffers are used. It
      is passed in the ancillary data.
      Bits (0-7) of the payload contain the vring index. Bit 8 is the
      invalid FD flag. This flag is set when there is no file descriptor
      in the ancillary data. This signals that polling will be used
      instead of waiting for the call.

 * VHOST_USER_SET_VRING_ERR

      Id: 14
      Equivalent ioctl: VHOST_SET_VRING_ERR
      Master payload: u64

      Set the event file descriptor to signal when error occurs. It
      is passed in the ancillary data.
      Bits (0-7) of the payload contain the vring index. Bit 8 is the
      invalid FD flag. This flag is set when there is no file descriptor
      in the ancillary data.