blob: 33588a03961f8910449d3e86e65116dbaf95eacb [file] [log] [blame]
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +02001Hyper-V Enlightenments
2======================
3
4
51. Description
6===============
7In some cases when implementing a hardware interface in software is slow, KVM
8implements its own paravirtualized interfaces. This works well for Linux as
9guest support for such features is added simultaneously with the feature itself.
10It may, however, be hard-to-impossible to add support for these interfaces to
11proprietary OSes, namely, Microsoft Windows.
12
13KVM on x86 implements Hyper-V Enlightenments for Windows guests. These features
14make Windows and Hyper-V guests think they're running on top of a Hyper-V
15compatible hypervisor and use Hyper-V specific features.
16
17
182. Setup
19=========
20No Hyper-V enlightenments are enabled by default by either KVM or QEMU. In
21QEMU, individual enlightenments can be enabled through CPU flags, e.g:
22
23 qemu-system-x86_64 --enable-kvm --cpu host,hv_relaxed,hv_vpindex,hv_time, ...
24
25Sometimes there are dependencies between enlightenments, QEMU is supposed to
26check that the supplied configuration is sane.
27
28When any set of the Hyper-V enlightenments is enabled, QEMU changes hypervisor
29identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification
30and features are kept in leaves 0x40000100..0x40000101.
31
32
333. Existing enlightenments
34===========================
35
363.1. hv-relaxed
37================
38This feature tells guest OS to disable watchdog timeouts as it is running on a
39hypervisor. It is known that some Windows versions will do this even when they
40see 'hypervisor' CPU flag.
41
423.2. hv-vapic
43==============
44Provides so-called VP Assist page MSR to guest allowing it to work with APIC
45more efficiently. In particular, this enlightenment allows paravirtualized
46(exit-less) EOI processing.
47
483.3. hv-spinlocks=xxx
49======================
50Enables paravirtualized spinlocks. The parameter indicates how many times
51spinlock acquisition should be attempted before indicating the situation to the
Vitaly Kuznetsovf701c082020-05-15 13:48:47 +020052hypervisor. A special value 0xffffffff indicates "never notify".
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +020053
543.4. hv-vpindex
55================
56Provides HV_X64_MSR_VP_INDEX (0x40000002) MSR to the guest which has Virtual
57processor index information. This enlightenment makes sense in conjunction with
58hv-synic, hv-stimer and other enlightenments which require the guest to know its
59Virtual Processor indices (e.g. when VP index needs to be passed in a
60hypercall).
61
623.5. hv-runtime
63================
64Provides HV_X64_MSR_VP_RUNTIME (0x40000010) MSR to the guest. The MSR keeps the
65virtual processor run time in 100ns units. This gives guest operating system an
66idea of how much time was 'stolen' from it (when the virtual CPU was preempted
67to perform some other work).
68
693.6. hv-crash
70==============
71Provides HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 (0x40000100..0x40000105) and
72HV_X64_MSR_CRASH_CTL (0x40000105) MSRs to the guest. These MSRs are written to
73by the guest when it crashes, HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 MSRs
74contain additional crash information. This information is outputted in QEMU log
75and through QAPI.
76Note: unlike under genuine Hyper-V, write to HV_X64_MSR_CRASH_CTL causes guest
77to shutdown. This effectively blocks crash dump generation by Windows.
78
793.7. hv-time
80=============
81Enables two Hyper-V-specific clocksources available to the guest: MSR-based
82Hyper-V clocksource (HV_X64_MSR_TIME_REF_COUNT, 0x40000020) and Reference TSC
83page (enabled via MSR HV_X64_MSR_REFERENCE_TSC, 0x40000021). Both clocksources
84are per-guest, Reference TSC page clocksource allows for exit-less time stamp
85readings. Using this enlightenment leads to significant speedup of all timestamp
86related operations.
87
883.8. hv-synic
89==============
90Enables Hyper-V Synthetic interrupt controller - an extension of a local APIC.
91When enabled, this enlightenment provides additional communication facilities
92to the guest: SynIC messages and Events. This is a pre-requisite for
93implementing VMBus devices (not yet in QEMU). Additionally, this enlightenment
94is needed to enable Hyper-V synthetic timers. SynIC is controlled through MSRs
95HV_X64_MSR_SCONTROL..HV_X64_MSR_EOM (0x40000080..0x40000084) and
96HV_X64_MSR_SINT0..HV_X64_MSR_SINT15 (0x40000090..0x4000009F)
97
98Requires: hv-vpindex
99
1003.9. hv-stimer
101===============
102Enables Hyper-V synthetic timers. There are four synthetic timers per virtual
103CPU controlled through HV_X64_MSR_STIMER0_CONFIG..HV_X64_MSR_STIMER3_COUNT
104(0x400000B0..0x400000B7) MSRs. These timers can work either in single-shot or
105periodic mode. It is known that certain Windows versions revert to using HPET
106(or even RTC when HPET is unavailable) extensively when this enlightenment is
107not provided; this can lead to significant CPU consumption, even when virtual
108CPU is idle.
109
110Requires: hv-vpindex, hv-synic, hv-time
111
1123.10. hv-tlbflush
113==================
114Enables paravirtualized TLB shoot-down mechanism. On x86 architecture, remote
115TLB flush procedure requires sending IPIs and waiting for other CPUs to perform
116local TLB flush. In virtualized environment some virtual CPUs may not even be
117scheduled at the time of the call and may not require flushing (or, flushing
118may be postponed until the virtual CPU is scheduled). hv-tlbflush enlightenment
119implements TLB shoot-down through hypervisor enabling the optimization.
120
121Requires: hv-vpindex
122
1233.11. hv-ipi
124=============
125Enables paravirtualized IPI send mechanism. HvCallSendSyntheticClusterIpi
126hypercall may target more than 64 virtual CPUs simultaneously, doing the same
127through APIC requires more than one access (and thus exit to the hypervisor).
128
129Requires: hv-vpindex
130
1313.12. hv-vendor-id=xxx
132=======================
133This changes Hyper-V identification in CPUID 0x40000000.EBX-EDX from the default
134"Microsoft Hv". The parameter should be no longer than 12 characters. According
135to the specification, guests shouldn't use this information and it is unknown
136if there is a Windows version which acts differently.
137Note: hv-vendor-id is not an enlightenment and thus doesn't enable Hyper-V
138identification when specified without some other enlightenment.
139
1403.13. hv-reset
141===============
142Provides HV_X64_MSR_RESET (0x40000003) MSR to the guest allowing it to reset
143itself by writing to it. Even when this MSR is enabled, it is not a recommended
144way for Windows to perform system reboot and thus it may not be used.
145
1463.14. hv-frequencies
147============================================
148Provides HV_X64_MSR_TSC_FREQUENCY (0x40000022) and HV_X64_MSR_APIC_FREQUENCY
149(0x40000023) allowing the guest to get its TSC/APIC frequencies without doing
150measurements.
151
1523.15 hv-reenlightenment
153========================
154The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
155enabled, it provides HV_X64_MSR_REENLIGHTENMENT_CONTROL (0x40000106),
156HV_X64_MSR_TSC_EMULATION_CONTROL (0x40000107)and HV_X64_MSR_TSC_EMULATION_STATUS
157(0x40000108) MSRs allowing the guest to get notified when TSC frequency changes
158(only happens on migration) and keep using old frequency (through emulation in
159the hypervisor) until it is ready to switch to the new one. This, in conjunction
160with hv-frequencies, allows Hyper-V on KVM to pass stable clocksource (Reference
161TSC page) to its own guests.
162
Vitaly Kuznetsov561dbb42021-03-19 13:38:01 +0100163Note, KVM doesn't fully support re-enlightenment notifications and doesn't
164emulate TSC accesses after migration so 'tsc-frequency=' CPU option also has to
165be specified to make migration succeed. The destination host has to either have
166the same TSC frequency or support TSC scaling CPU feature.
167
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +0200168Recommended: hv-frequencies
169
1703.16. hv-evmcs
171===============
172The enlightenment is nested specific, it targets Hyper-V on KVM guests. When
Vitaly Kuznetsov07454e22021-06-08 14:08:11 +0200173enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +0200174implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V)
175hypervisors making L2 exits to the hypervisor faster. The feature is Intel-only.
176Note: some virtualization features (e.g. Posted Interrupts) are disabled when
177hv-evmcs is enabled. It may make sense to measure your nested workload with and
178without the feature to find out if enabling it is beneficial.
179
180Requires: hv-vapic
181
Vitaly Kuznetsov128531d2019-05-17 16:19:24 +02001823.17. hv-stimer-direct
183=======================
184Hyper-V specification allows synthetic timer operation in two modes: "classic",
185when expiration event is delivered as SynIC message and "direct", when the event
186is delivered via normal interrupt. It is known that nested Hyper-V can only
187use synthetic timers in direct mode and thus 'hv-stimer-direct' needs to be
188enabled.
189
190Requires: hv-vpindex, hv-synic, hv-time, hv-stimer
191
Vitaly Kuznetsove1f9a8e2021-09-02 11:35:28 +02001923.18. hv-avic (hv-apicv)
193=======================
194The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled.
195Normally, Hyper-V SynIC disables these hardware feature and suggests the guest
196to use paravirtualized AutoEOI feature.
197Note: enabling this feature on old hardware (without APICv/AVIC support) may
Stefan Weilb980c1a2021-11-17 22:07:02 +0100198have negative effect on guest's performance.
Vitaly Kuznetsove1f9a8e2021-09-02 11:35:28 +0200199
2003.19. hv-no-nonarch-coresharing=on/off/auto
Vitaly Kuznetsov30d6ff62019-10-18 18:39:08 +0200201===========================================
202This enlightenment tells guest OS that virtual processors will never share a
203physical core unless they are reported as sibling SMT threads. This information
204is required by Windows and Hyper-V guests to properly mitigate SMT related CPU
205vulnerabilities.
206When the option is set to 'auto' QEMU will enable the feature only when KVM
207reports that non-architectural coresharing is impossible, this means that
208hyper-threading is not supported or completely disabled on the host. This
209setting also prevents migration as SMT settings on the destination may differ.
210When the option is set to 'on' QEMU will always enable the feature, regardless
211of host setup. To keep guests secure, this can only be used in conjunction with
212exposing correct vCPU topology and vCPU pinning.
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +0200213
Vitaly Kuznetsovaf7228b2021-09-02 11:35:29 +02002143.20. hv-version-id-{build,major,minor,spack,sbranch,snumber}
215=============================================================
216This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the
Vitaly Kuznetsovf701ece2021-09-02 11:35:30 +0200217default (WS2016).
Vitaly Kuznetsovaf7228b2021-09-02 11:35:29 +0200218- hv-version-id-build sets 'Build Number' (32 bits)
219- hv-version-id-major sets 'Major Version' (16 bits)
220- hv-version-id-minor sets 'Minor Version' (16 bits)
221- hv-version-id-spack sets 'Service Pack' (32 bits)
222- hv-version-id-sbranch sets 'Service Branch' (8 bits)
223- hv-version-id-snumber sets 'Service Number' (24 bits)
224
225Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V
226identification when specified without any other enlightenments.
227
Jon Doron73d24072022-02-16 12:24:59 +02002283.21. hv-syndbg
229===============
230Enables Hyper-V synthetic debugger interface, this is a special interface used
231by Windows Kernel debugger to send the packets through, rather than sending
232them via serial/network .
233When enabled, this enlightenment provides additional communication facilities
234to the guest: SynDbg messages.
235This new communication is used by Windows Kernel debugger rather than sending
236packets via serial/network, adding significant performance boost over the other
237comm channels.
238This enlightenment requires a VMBus device (-device vmbus-bridge,irq=15)
239and the follow enlightenments to work:
240hv-relaxed,hv_time,hv-vapic,hv-vpindex,hv-synic,hv-runtime,hv-stimer
241
242
Vitaly Kuznetsov70367f02021-09-02 11:35:26 +02002434. Supplementary features
244=========================
245
2464.1. hv-passthrough
247===================
Vitaly Kuznetsove48ddcc2019-05-17 16:19:20 +0200248In some cases (e.g. during development) it may make sense to use QEMU in
249'pass-through' mode and give Windows guests all enlightenments currently
250supported by KVM. This pass-through mode is enabled by "hv-passthrough" CPU
251flag.
Vitaly Kuznetsov00c1b312021-06-08 14:08:10 +0200252Note: "hv-passthrough" flag only enables enlightenments which are known to QEMU
253(have corresponding "hv-*" flag) and copies "hv-spinlocks="/"hv-vendor-id="
254values from KVM to QEMU. "hv-passthrough" overrides all other "hv-*" settings on
255the command line. Also, enabling this flag effectively prevents migration as the
256list of enabled enlightenments may differ between target and destination hosts.
Vitaly Kuznetsove48ddcc2019-05-17 16:19:20 +0200257
Vitaly Kuznetsov70367f02021-09-02 11:35:26 +02002584.2. hv-enforce-cpuid
259=====================
260By default, KVM allows the guest to use all currently supported Hyper-V
261enlightenments when Hyper-V CPUID interface was exposed, regardless of if
262some features were not announced in guest visible CPUIDs. 'hv-enforce-cpuid'
263feature alters this behavior and only allows the guest to use exposed Hyper-V
264enlightenments.
Vitaly Kuznetsove48ddcc2019-05-17 16:19:20 +0200265
Vitaly Kuznetsov70367f02021-09-02 11:35:26 +0200266
2675. Useful links
Vitaly Kuznetsovfb19f722019-05-17 16:19:19 +0200268================
269Hyper-V Top Level Functional specification and other information:
270https://github.com/MicrosoftDocs/Virtualization-Documentation