docs/system/i386/sgx.rst - qemu - Git at Google

 Software Guard eXtensions (SGX)
 ===============================

 Overview
 --------

 Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
 for memory accesses in order to provide security accesses for sensitive
 applications and data. SGX allows an application to use its particular
 address space as an *enclave*, which is a protected area provides confidentiality
 and integrity even in the presence of privileged malware. Accesses to the
 enclave memory area from any software not resident in the enclave are prevented,
 including those from privileged software.

 Virtual SGX
 -----------

 SGX feature is exposed to guest via SGX CPUID. Looking at SGX CPUID, we can
 report the same CPUID info to guest as on host for most of SGX CPUID. With
 reporting the same CPUID guest is able to use full capacity of SGX, and KVM
 doesn't need to emulate those info.

 The guest's EPC base and size are determined by QEMU, and KVM needs QEMU to
 notify such info to it before it can initialize SGX for guest.

 Virtual EPC
 ~~~~~~~~~~~

 By default, QEMU does not assign EPC to a VM, i.e. fully enabling SGX in a VM
 requires explicit allocation of EPC to the VM. Similar to other specialized
 memory types, e.g. hugetlbfs, EPC is exposed as a memory backend.

 SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be realized
 prior to realizing the vCPUs themselves, which occurs long before generic
 devices are parsed and realized.  This limitation means that EPC does not
 require -maxmem as EPC is not treated as {cold,hot}plugged memory.

 QEMU does not artificially restrict the number of EPC sections exposed to a
 guest, e.g. QEMU will happily allow you to create 64 1M EPC sections. Be aware
 that some kernels may not recognize all EPC sections, e.g. the Linux SGX driver
 is hardwired to support only 8 EPC sections.

 The following QEMU snippet creates two EPC sections, with 64M pre-allocated
 to the VM and an additional 28M mapped but not allocated::

  -object memory-backend-epc,id=mem1,size=64M,prealloc=on \
  -object memory-backend-epc,id=mem2,size=28M \
  -M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2

 Note:

 The size and location of the virtual EPC are far less restricted compared
 to physical EPC. Because physical EPC is protected via range registers,
 the size of the physical EPC must be a power of two (though software sees
 a subset of the full EPC, e.g. 92M or 128M) and the EPC must be naturally
 aligned.  KVM SGX's virtual EPC is purely a software construct and only
 requires the size and location to be page aligned. QEMU enforces the EPC
 size is a multiple of 4k and will ensure the base of the EPC is 4k aligned.
 To simplify the implementation, EPC is always located above 4g in the guest
 physical address space.

 Migration
 ~~~~~~~~~

 QEMU/KVM doesn't prevent live migrating SGX VMs, although from hardware's
 perspective, SGX doesn't support live migration, since both EPC and the SGX
 key hierarchy are bound to the physical platform. However live migration
 can be supported in the sense if guest software stack can support recreating
 enclaves when it suffers sudden lose of EPC; and if guest enclaves can detect
 SGX keys being changed, and handle gracefully. For instance, when ERESUME fails
 with #PF.SGX, guest software can gracefully detect it and recreate enclaves;
 and when enclave fails to unseal sensitive information from outside, it can
 detect such error and sensitive information can be provisioned to it again.

 CPUID
 ~~~~~

 Due to its myriad dependencies, SGX is currently not listed as supported
 in any of QEMU's built-in CPU configuration. To expose SGX (and SGX Launch
 Control) to a guest, you must either use ``-cpu host`` to pass-through the
 host CPU model, or explicitly enable SGX when using a built-in CPU model,
 e.g. via ``-cpu <model>,+sgx`` or ``-cpu <model>,+sgx,+sgxlc``.

 All SGX sub-features enumerated through CPUID, e.g. SGX2, MISCSELECT,
 ATTRIBUTES, etc... can be restricted via CPUID flags. Be aware that enforcing
 restriction of MISCSELECT, ATTRIBUTES and XFRM requires intercepting ECREATE,
 i.e. may marginally reduce SGX performance in the guest. All SGX sub-features
 controlled via -cpu are prefixed with "sgx", e.g.::

   $ qemu-system-x86_64 -cpu help | xargs printf "%s\n" | grep sgx
   sgx
   sgx-debug
   sgx-encls-c
   sgx-enclv
   sgx-exinfo
   sgx-kss
   sgx-mode64
   sgx-provisionkey
   sgx-tokenkey
   sgx1
   sgx2
   sgxlc

 The following QEMU snippet passes through the host CPU but restricts access to
 the provision and EINIT token keys::

  -cpu host,-sgx-provisionkey,-sgx-tokenkey

 SGX sub-features cannot be emulated, i.e. sub-features that are not present
 in hardware cannot be forced on via '-cpu'.

 Virtualize SGX Launch Control
 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

 QEMU SGX support for Launch Control (LC) is passive, in the sense that it
 does not actively change the LC configuration.  QEMU SGX provides the user
 the ability to set/clear the CPUID flag (and by extension the associated
 IA32_FEATURE_CONTROL MSR bit in fw_cfg) and saves/restores the LE Hash MSRs
 when getting/putting guest state, but QEMU does not add new controls to
 directly modify the LC configuration.  Similar to hardware behavior, locking
 the LC configuration to a non-Intel value is left to guest firmware.  Unlike
 host bios setting for SGX launch control(LC), there is no special bios setting
 for SGX guest by our design. If host is in locked mode, we can still allow
 creating VM with SGX.

 Feature Control
 ~~~~~~~~~~~~~~~

 QEMU SGX updates the ``etc/msr_feature_control`` fw_cfg entry to set the SGX
 (bit 18) and SGX LC (bit 17) flags based on their respective CPUID support,
 i.e. existing guest firmware will automatically set SGX and SGX LC accordingly,
 assuming said firmware supports fw_cfg.msr_feature_control.

 Launching a guest
 -----------------

 To launch a SGX guest:

 .. parsed-literal::

   |qemu_system_x86| \\
    -cpu host,+sgx-provisionkey \\
    -object memory-backend-epc,id=mem1,size=64M,prealloc=on \\
    -M sgx-epc.0.memdev=mem1,sgx-epc.0.node=0

 Utilizing SGX in the guest requires a kernel/OS with SGX support.
 The support can be determined in guest by::

   $ grep sgx /proc/cpuinfo

 and SGX epc info by::

   $ dmesg | grep sgx
   [    0.182807] sgx: EPC section 0x140000000-0x143ffffff
   [    0.183695] sgx: [Firmware Bug]: Unable to map EPC section to online node. Fallback to the NUMA node 0.

 To launch a SGX numa guest:

 .. parsed-literal::

   |qemu_system_x86| \\
    -cpu host,+sgx-provisionkey \\
    -object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \\
    -object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \\
    -numa node,nodeid=0,cpus=0-1,memdev=node0 \\
    -object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \\
    -object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \\
    -numa node,nodeid=1,cpus=2-3,memdev=node1 \\
    -M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1

 and SGX epc numa info by::

   $ dmesg | grep sgx
   [    0.369937] sgx: EPC section 0x180000000-0x183ffffff
   [    0.370259] sgx: EPC section 0x184000000-0x185bfffff

   $ dmesg | grep SRAT
   [    0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff]
   [    0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff]

 References
 ----------

 - `SGX Homepage <https://software.intel.com/sgx>`__

 - `SGX SDK <https://github.com/intel/linux-sgx.git>`__

 - SGX specification: Intel SDM Volume 3
	Software Guard eXtensions (SGX)
	===============================

	Overview
	--------

	Intel Software Guard eXtensions (SGX) is a set of instructions and mechanisms
	for memory accesses in order to provide security accesses for sensitive
	applications and data. SGX allows an application to use its particular
	address space as an enclave, which is a protected area provides confidentiality
	and integrity even in the presence of privileged malware. Accesses to the
	enclave memory area from any software not resident in the enclave are prevented,
	including those from privileged software.

	Virtual SGX
	-----------

	SGX feature is exposed to guest via SGX CPUID. Looking at SGX CPUID, we can
	report the same CPUID info to guest as on host for most of SGX CPUID. With
	reporting the same CPUID guest is able to use full capacity of SGX, and KVM
	doesn't need to emulate those info.

	The guest's EPC base and size are determined by QEMU, and KVM needs QEMU to
	notify such info to it before it can initialize SGX for guest.

	Virtual EPC
	~~~~~~~~~~~

	By default, QEMU does not assign EPC to a VM, i.e. fully enabling SGX in a VM
	requires explicit allocation of EPC to the VM. Similar to other specialized
	memory types, e.g. hugetlbfs, EPC is exposed as a memory backend.

	SGX EPC is enumerated through CPUID, i.e. EPC "devices" need to be realized
	prior to realizing the vCPUs themselves, which occurs long before generic
	devices are parsed and realized. This limitation means that EPC does not
	require -maxmem as EPC is not treated as {cold,hot}plugged memory.

	QEMU does not artificially restrict the number of EPC sections exposed to a
	guest, e.g. QEMU will happily allow you to create 64 1M EPC sections. Be aware
	that some kernels may not recognize all EPC sections, e.g. the Linux SGX driver
	is hardwired to support only 8 EPC sections.

	The following QEMU snippet creates two EPC sections, with 64M pre-allocated
	to the VM and an additional 28M mapped but not allocated::

	-object memory-backend-epc,id=mem1,size=64M,prealloc=on \
	-object memory-backend-epc,id=mem2,size=28M \
	-M sgx-epc.0.memdev=mem1,sgx-epc.1.memdev=mem2

	Note:

	The size and location of the virtual EPC are far less restricted compared
	to physical EPC. Because physical EPC is protected via range registers,
	the size of the physical EPC must be a power of two (though software sees
	a subset of the full EPC, e.g. 92M or 128M) and the EPC must be naturally
	aligned. KVM SGX's virtual EPC is purely a software construct and only
	requires the size and location to be page aligned. QEMU enforces the EPC
	size is a multiple of 4k and will ensure the base of the EPC is 4k aligned.
	To simplify the implementation, EPC is always located above 4g in the guest
	physical address space.

	Migration
	~~~~~~~~~

	QEMU/KVM doesn't prevent live migrating SGX VMs, although from hardware's
	perspective, SGX doesn't support live migration, since both EPC and the SGX
	key hierarchy are bound to the physical platform. However live migration
	can be supported in the sense if guest software stack can support recreating
	enclaves when it suffers sudden lose of EPC; and if guest enclaves can detect
	SGX keys being changed, and handle gracefully. For instance, when ERESUME fails
	with #PF.SGX, guest software can gracefully detect it and recreate enclaves;
	and when enclave fails to unseal sensitive information from outside, it can
	detect such error and sensitive information can be provisioned to it again.

	CPUID
	~~~~~

	Due to its myriad dependencies, SGX is currently not listed as supported
	in any of QEMU's built-in CPU configuration. To expose SGX (and SGX Launch
	Control) to a guest, you must either use ``-cpu host`` to pass-through the
	host CPU model, or explicitly enable SGX when using a built-in CPU model,
	e.g. via ``-cpu <model>,+sgx`` or ``-cpu <model>,+sgx,+sgxlc``.

	All SGX sub-features enumerated through CPUID, e.g. SGX2, MISCSELECT,
	ATTRIBUTES, etc... can be restricted via CPUID flags. Be aware that enforcing
	restriction of MISCSELECT, ATTRIBUTES and XFRM requires intercepting ECREATE,
	i.e. may marginally reduce SGX performance in the guest. All SGX sub-features
	controlled via -cpu are prefixed with "sgx", e.g.::

	$ qemu-system-x86_64 -cpu help \| xargs printf "%s\n" \| grep sgx
	sgx
	sgx-debug
	sgx-encls-c
	sgx-enclv
	sgx-exinfo
	sgx-kss
	sgx-mode64
	sgx-provisionkey
	sgx-tokenkey
	sgx1
	sgx2
	sgxlc

	The following QEMU snippet passes through the host CPU but restricts access to
	the provision and EINIT token keys::

	-cpu host,-sgx-provisionkey,-sgx-tokenkey

	SGX sub-features cannot be emulated, i.e. sub-features that are not present
	in hardware cannot be forced on via '-cpu'.

	Virtualize SGX Launch Control
	~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

	QEMU SGX support for Launch Control (LC) is passive, in the sense that it
	does not actively change the LC configuration. QEMU SGX provides the user
	the ability to set/clear the CPUID flag (and by extension the associated
	IA32_FEATURE_CONTROL MSR bit in fw_cfg) and saves/restores the LE Hash MSRs
	when getting/putting guest state, but QEMU does not add new controls to
	directly modify the LC configuration. Similar to hardware behavior, locking
	the LC configuration to a non-Intel value is left to guest firmware. Unlike
	host bios setting for SGX launch control(LC), there is no special bios setting
	for SGX guest by our design. If host is in locked mode, we can still allow
	creating VM with SGX.

	Feature Control
	~~~~~~~~~~~~~~~

	QEMU SGX updates the ``etc/msr_feature_control`` fw_cfg entry to set the SGX
	(bit 18) and SGX LC (bit 17) flags based on their respective CPUID support,
	i.e. existing guest firmware will automatically set SGX and SGX LC accordingly,
	assuming said firmware supports fw_cfg.msr_feature_control.

	Launching a guest
	-----------------

	To launch a SGX guest:

	.. parsed-literal::

	\|qemu_system_x86\| \\
	-cpu host,+sgx-provisionkey \\
	-object memory-backend-epc,id=mem1,size=64M,prealloc=on \\
	-M sgx-epc.0.memdev=mem1,sgx-epc.0.node=0

	Utilizing SGX in the guest requires a kernel/OS with SGX support.
	The support can be determined in guest by::

	$ grep sgx /proc/cpuinfo

	and SGX epc info by::

	$ dmesg \| grep sgx
	[ 0.182807] sgx: EPC section 0x140000000-0x143ffffff
	[ 0.183695] sgx: [Firmware Bug]: Unable to map EPC section to online node. Fallback to the NUMA node 0.

	To launch a SGX numa guest:

	.. parsed-literal::

	\|qemu_system_x86\| \\
	-cpu host,+sgx-provisionkey \\
	-object memory-backend-ram,size=2G,host-nodes=0,policy=bind,id=node0 \\
	-object memory-backend-epc,id=mem0,size=64M,prealloc=on,host-nodes=0,policy=bind \\
	-numa node,nodeid=0,cpus=0-1,memdev=node0 \\
	-object memory-backend-ram,size=2G,host-nodes=1,policy=bind,id=node1 \\
	-object memory-backend-epc,id=mem1,size=28M,prealloc=on,host-nodes=1,policy=bind \\
	-numa node,nodeid=1,cpus=2-3,memdev=node1 \\
	-M sgx-epc.0.memdev=mem0,sgx-epc.0.node=0,sgx-epc.1.memdev=mem1,sgx-epc.1.node=1

	and SGX epc numa info by::

	$ dmesg \| grep sgx
	[ 0.369937] sgx: EPC section 0x180000000-0x183ffffff
	[ 0.370259] sgx: EPC section 0x184000000-0x185bfffff

	$ dmesg \| grep SRAT
	[ 0.009981] ACPI: SRAT: Node 0 PXM 0 [mem 0x180000000-0x183ffffff]
	[ 0.009982] ACPI: SRAT: Node 1 PXM 1 [mem 0x184000000-0x185bfffff]

	References
	----------

	- `SGX Homepage <https://software.intel.com/sgx>`__

	- `SGX SDK <https://github.com/intel/linux-sgx.git>`__

	- SGX specification: Intel SDM Volume 3