Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 1 | Hyper-V Enlightenments |
| 2 | ====================== |
| 3 | |
| 4 | |
| 5 | 1. Description |
| 6 | =============== |
| 7 | In some cases when implementing a hardware interface in software is slow, KVM |
| 8 | implements its own paravirtualized interfaces. This works well for Linux as |
| 9 | guest support for such features is added simultaneously with the feature itself. |
| 10 | It may, however, be hard-to-impossible to add support for these interfaces to |
| 11 | proprietary OSes, namely, Microsoft Windows. |
| 12 | |
| 13 | KVM on x86 implements Hyper-V Enlightenments for Windows guests. These features |
| 14 | make Windows and Hyper-V guests think they're running on top of a Hyper-V |
| 15 | compatible hypervisor and use Hyper-V specific features. |
| 16 | |
| 17 | |
| 18 | 2. Setup |
| 19 | ========= |
| 20 | No Hyper-V enlightenments are enabled by default by either KVM or QEMU. In |
| 21 | QEMU, individual enlightenments can be enabled through CPU flags, e.g: |
| 22 | |
| 23 | qemu-system-x86_64 --enable-kvm --cpu host,hv_relaxed,hv_vpindex,hv_time, ... |
| 24 | |
| 25 | Sometimes there are dependencies between enlightenments, QEMU is supposed to |
| 26 | check that the supplied configuration is sane. |
| 27 | |
| 28 | When any set of the Hyper-V enlightenments is enabled, QEMU changes hypervisor |
| 29 | identification (CPUID 0x40000000..0x4000000A) to Hyper-V. KVM identification |
| 30 | and features are kept in leaves 0x40000100..0x40000101. |
| 31 | |
| 32 | |
| 33 | 3. Existing enlightenments |
| 34 | =========================== |
| 35 | |
| 36 | 3.1. hv-relaxed |
| 37 | ================ |
| 38 | This feature tells guest OS to disable watchdog timeouts as it is running on a |
| 39 | hypervisor. It is known that some Windows versions will do this even when they |
| 40 | see 'hypervisor' CPU flag. |
| 41 | |
| 42 | 3.2. hv-vapic |
| 43 | ============== |
| 44 | Provides so-called VP Assist page MSR to guest allowing it to work with APIC |
| 45 | more efficiently. In particular, this enlightenment allows paravirtualized |
| 46 | (exit-less) EOI processing. |
| 47 | |
| 48 | 3.3. hv-spinlocks=xxx |
| 49 | ====================== |
| 50 | Enables paravirtualized spinlocks. The parameter indicates how many times |
| 51 | spinlock acquisition should be attempted before indicating the situation to the |
Vitaly Kuznetsov | f701c08 | 2020-05-15 13:48:47 +0200 | [diff] [blame] | 52 | hypervisor. A special value 0xffffffff indicates "never notify". |
Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 53 | |
| 54 | 3.4. hv-vpindex |
| 55 | ================ |
| 56 | Provides HV_X64_MSR_VP_INDEX (0x40000002) MSR to the guest which has Virtual |
| 57 | processor index information. This enlightenment makes sense in conjunction with |
| 58 | hv-synic, hv-stimer and other enlightenments which require the guest to know its |
| 59 | Virtual Processor indices (e.g. when VP index needs to be passed in a |
| 60 | hypercall). |
| 61 | |
| 62 | 3.5. hv-runtime |
| 63 | ================ |
| 64 | Provides HV_X64_MSR_VP_RUNTIME (0x40000010) MSR to the guest. The MSR keeps the |
| 65 | virtual processor run time in 100ns units. This gives guest operating system an |
| 66 | idea of how much time was 'stolen' from it (when the virtual CPU was preempted |
| 67 | to perform some other work). |
| 68 | |
| 69 | 3.6. hv-crash |
| 70 | ============== |
| 71 | Provides HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 (0x40000100..0x40000105) and |
| 72 | HV_X64_MSR_CRASH_CTL (0x40000105) MSRs to the guest. These MSRs are written to |
| 73 | by the guest when it crashes, HV_X64_MSR_CRASH_P0..HV_X64_MSR_CRASH_P5 MSRs |
| 74 | contain additional crash information. This information is outputted in QEMU log |
| 75 | and through QAPI. |
| 76 | Note: unlike under genuine Hyper-V, write to HV_X64_MSR_CRASH_CTL causes guest |
| 77 | to shutdown. This effectively blocks crash dump generation by Windows. |
| 78 | |
| 79 | 3.7. hv-time |
| 80 | ============= |
| 81 | Enables two Hyper-V-specific clocksources available to the guest: MSR-based |
| 82 | Hyper-V clocksource (HV_X64_MSR_TIME_REF_COUNT, 0x40000020) and Reference TSC |
| 83 | page (enabled via MSR HV_X64_MSR_REFERENCE_TSC, 0x40000021). Both clocksources |
| 84 | are per-guest, Reference TSC page clocksource allows for exit-less time stamp |
| 85 | readings. Using this enlightenment leads to significant speedup of all timestamp |
| 86 | related operations. |
| 87 | |
| 88 | 3.8. hv-synic |
| 89 | ============== |
| 90 | Enables Hyper-V Synthetic interrupt controller - an extension of a local APIC. |
| 91 | When enabled, this enlightenment provides additional communication facilities |
| 92 | to the guest: SynIC messages and Events. This is a pre-requisite for |
| 93 | implementing VMBus devices (not yet in QEMU). Additionally, this enlightenment |
| 94 | is needed to enable Hyper-V synthetic timers. SynIC is controlled through MSRs |
| 95 | HV_X64_MSR_SCONTROL..HV_X64_MSR_EOM (0x40000080..0x40000084) and |
| 96 | HV_X64_MSR_SINT0..HV_X64_MSR_SINT15 (0x40000090..0x4000009F) |
| 97 | |
| 98 | Requires: hv-vpindex |
| 99 | |
| 100 | 3.9. hv-stimer |
| 101 | =============== |
| 102 | Enables Hyper-V synthetic timers. There are four synthetic timers per virtual |
| 103 | CPU controlled through HV_X64_MSR_STIMER0_CONFIG..HV_X64_MSR_STIMER3_COUNT |
| 104 | (0x400000B0..0x400000B7) MSRs. These timers can work either in single-shot or |
| 105 | periodic mode. It is known that certain Windows versions revert to using HPET |
| 106 | (or even RTC when HPET is unavailable) extensively when this enlightenment is |
| 107 | not provided; this can lead to significant CPU consumption, even when virtual |
| 108 | CPU is idle. |
| 109 | |
| 110 | Requires: hv-vpindex, hv-synic, hv-time |
| 111 | |
| 112 | 3.10. hv-tlbflush |
| 113 | ================== |
| 114 | Enables paravirtualized TLB shoot-down mechanism. On x86 architecture, remote |
| 115 | TLB flush procedure requires sending IPIs and waiting for other CPUs to perform |
| 116 | local TLB flush. In virtualized environment some virtual CPUs may not even be |
| 117 | scheduled at the time of the call and may not require flushing (or, flushing |
| 118 | may be postponed until the virtual CPU is scheduled). hv-tlbflush enlightenment |
| 119 | implements TLB shoot-down through hypervisor enabling the optimization. |
| 120 | |
| 121 | Requires: hv-vpindex |
| 122 | |
| 123 | 3.11. hv-ipi |
| 124 | ============= |
| 125 | Enables paravirtualized IPI send mechanism. HvCallSendSyntheticClusterIpi |
| 126 | hypercall may target more than 64 virtual CPUs simultaneously, doing the same |
| 127 | through APIC requires more than one access (and thus exit to the hypervisor). |
| 128 | |
| 129 | Requires: hv-vpindex |
| 130 | |
| 131 | 3.12. hv-vendor-id=xxx |
| 132 | ======================= |
| 133 | This changes Hyper-V identification in CPUID 0x40000000.EBX-EDX from the default |
| 134 | "Microsoft Hv". The parameter should be no longer than 12 characters. According |
| 135 | to the specification, guests shouldn't use this information and it is unknown |
| 136 | if there is a Windows version which acts differently. |
| 137 | Note: hv-vendor-id is not an enlightenment and thus doesn't enable Hyper-V |
| 138 | identification when specified without some other enlightenment. |
| 139 | |
| 140 | 3.13. hv-reset |
| 141 | =============== |
| 142 | Provides HV_X64_MSR_RESET (0x40000003) MSR to the guest allowing it to reset |
| 143 | itself by writing to it. Even when this MSR is enabled, it is not a recommended |
| 144 | way for Windows to perform system reboot and thus it may not be used. |
| 145 | |
| 146 | 3.14. hv-frequencies |
| 147 | ============================================ |
| 148 | Provides HV_X64_MSR_TSC_FREQUENCY (0x40000022) and HV_X64_MSR_APIC_FREQUENCY |
| 149 | (0x40000023) allowing the guest to get its TSC/APIC frequencies without doing |
| 150 | measurements. |
| 151 | |
| 152 | 3.15 hv-reenlightenment |
| 153 | ======================== |
| 154 | The enlightenment is nested specific, it targets Hyper-V on KVM guests. When |
| 155 | enabled, it provides HV_X64_MSR_REENLIGHTENMENT_CONTROL (0x40000106), |
| 156 | HV_X64_MSR_TSC_EMULATION_CONTROL (0x40000107)and HV_X64_MSR_TSC_EMULATION_STATUS |
| 157 | (0x40000108) MSRs allowing the guest to get notified when TSC frequency changes |
| 158 | (only happens on migration) and keep using old frequency (through emulation in |
| 159 | the hypervisor) until it is ready to switch to the new one. This, in conjunction |
| 160 | with hv-frequencies, allows Hyper-V on KVM to pass stable clocksource (Reference |
| 161 | TSC page) to its own guests. |
| 162 | |
Vitaly Kuznetsov | 561dbb4 | 2021-03-19 13:38:01 +0100 | [diff] [blame] | 163 | Note, KVM doesn't fully support re-enlightenment notifications and doesn't |
| 164 | emulate TSC accesses after migration so 'tsc-frequency=' CPU option also has to |
| 165 | be specified to make migration succeed. The destination host has to either have |
| 166 | the same TSC frequency or support TSC scaling CPU feature. |
| 167 | |
Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 168 | Recommended: hv-frequencies |
| 169 | |
| 170 | 3.16. hv-evmcs |
| 171 | =============== |
| 172 | The enlightenment is nested specific, it targets Hyper-V on KVM guests. When |
Vitaly Kuznetsov | 07454e2 | 2021-06-08 14:08:11 +0200 | [diff] [blame] | 173 | enabled, it provides Enlightened VMCS version 1 feature to the guest. The feature |
Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 174 | implements paravirtualized protocol between L0 (KVM) and L1 (Hyper-V) |
| 175 | hypervisors making L2 exits to the hypervisor faster. The feature is Intel-only. |
| 176 | Note: some virtualization features (e.g. Posted Interrupts) are disabled when |
| 177 | hv-evmcs is enabled. It may make sense to measure your nested workload with and |
| 178 | without the feature to find out if enabling it is beneficial. |
| 179 | |
| 180 | Requires: hv-vapic |
| 181 | |
Vitaly Kuznetsov | 128531d | 2019-05-17 16:19:24 +0200 | [diff] [blame] | 182 | 3.17. hv-stimer-direct |
| 183 | ======================= |
| 184 | Hyper-V specification allows synthetic timer operation in two modes: "classic", |
| 185 | when expiration event is delivered as SynIC message and "direct", when the event |
| 186 | is delivered via normal interrupt. It is known that nested Hyper-V can only |
| 187 | use synthetic timers in direct mode and thus 'hv-stimer-direct' needs to be |
| 188 | enabled. |
| 189 | |
| 190 | Requires: hv-vpindex, hv-synic, hv-time, hv-stimer |
| 191 | |
Vitaly Kuznetsov | e1f9a8e | 2021-09-02 11:35:28 +0200 | [diff] [blame] | 192 | 3.18. hv-avic (hv-apicv) |
| 193 | ======================= |
| 194 | The enlightenment allows to use Hyper-V SynIC with hardware APICv/AVIC enabled. |
| 195 | Normally, Hyper-V SynIC disables these hardware feature and suggests the guest |
| 196 | to use paravirtualized AutoEOI feature. |
| 197 | Note: enabling this feature on old hardware (without APICv/AVIC support) may |
Stefan Weil | b980c1a | 2021-11-17 22:07:02 +0100 | [diff] [blame] | 198 | have negative effect on guest's performance. |
Vitaly Kuznetsov | e1f9a8e | 2021-09-02 11:35:28 +0200 | [diff] [blame] | 199 | |
| 200 | 3.19. hv-no-nonarch-coresharing=on/off/auto |
Vitaly Kuznetsov | 30d6ff6 | 2019-10-18 18:39:08 +0200 | [diff] [blame] | 201 | =========================================== |
| 202 | This enlightenment tells guest OS that virtual processors will never share a |
| 203 | physical core unless they are reported as sibling SMT threads. This information |
| 204 | is required by Windows and Hyper-V guests to properly mitigate SMT related CPU |
| 205 | vulnerabilities. |
| 206 | When the option is set to 'auto' QEMU will enable the feature only when KVM |
| 207 | reports that non-architectural coresharing is impossible, this means that |
| 208 | hyper-threading is not supported or completely disabled on the host. This |
| 209 | setting also prevents migration as SMT settings on the destination may differ. |
| 210 | When the option is set to 'on' QEMU will always enable the feature, regardless |
| 211 | of host setup. To keep guests secure, this can only be used in conjunction with |
| 212 | exposing correct vCPU topology and vCPU pinning. |
Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 213 | |
Vitaly Kuznetsov | af7228b | 2021-09-02 11:35:29 +0200 | [diff] [blame] | 214 | 3.20. hv-version-id-{build,major,minor,spack,sbranch,snumber} |
| 215 | ============================================================= |
| 216 | This changes Hyper-V version identification in CPUID 0x40000002.EAX-EDX from the |
Vitaly Kuznetsov | f701ece | 2021-09-02 11:35:30 +0200 | [diff] [blame] | 217 | default (WS2016). |
Vitaly Kuznetsov | af7228b | 2021-09-02 11:35:29 +0200 | [diff] [blame] | 218 | - hv-version-id-build sets 'Build Number' (32 bits) |
| 219 | - hv-version-id-major sets 'Major Version' (16 bits) |
| 220 | - hv-version-id-minor sets 'Minor Version' (16 bits) |
| 221 | - hv-version-id-spack sets 'Service Pack' (32 bits) |
| 222 | - hv-version-id-sbranch sets 'Service Branch' (8 bits) |
| 223 | - hv-version-id-snumber sets 'Service Number' (24 bits) |
| 224 | |
| 225 | Note: hv-version-id-* are not enlightenments and thus don't enable Hyper-V |
| 226 | identification when specified without any other enlightenments. |
| 227 | |
Jon Doron | 73d2407 | 2022-02-16 12:24:59 +0200 | [diff] [blame] | 228 | 3.21. hv-syndbg |
| 229 | =============== |
| 230 | Enables Hyper-V synthetic debugger interface, this is a special interface used |
| 231 | by Windows Kernel debugger to send the packets through, rather than sending |
| 232 | them via serial/network . |
| 233 | When enabled, this enlightenment provides additional communication facilities |
| 234 | to the guest: SynDbg messages. |
| 235 | This new communication is used by Windows Kernel debugger rather than sending |
| 236 | packets via serial/network, adding significant performance boost over the other |
| 237 | comm channels. |
| 238 | This enlightenment requires a VMBus device (-device vmbus-bridge,irq=15) |
| 239 | and the follow enlightenments to work: |
| 240 | hv-relaxed,hv_time,hv-vapic,hv-vpindex,hv-synic,hv-runtime,hv-stimer |
| 241 | |
| 242 | |
Vitaly Kuznetsov | 70367f0 | 2021-09-02 11:35:26 +0200 | [diff] [blame] | 243 | 4. Supplementary features |
| 244 | ========================= |
| 245 | |
| 246 | 4.1. hv-passthrough |
| 247 | =================== |
Vitaly Kuznetsov | e48ddcc | 2019-05-17 16:19:20 +0200 | [diff] [blame] | 248 | In some cases (e.g. during development) it may make sense to use QEMU in |
| 249 | 'pass-through' mode and give Windows guests all enlightenments currently |
| 250 | supported by KVM. This pass-through mode is enabled by "hv-passthrough" CPU |
| 251 | flag. |
Vitaly Kuznetsov | 00c1b31 | 2021-06-08 14:08:10 +0200 | [diff] [blame] | 252 | Note: "hv-passthrough" flag only enables enlightenments which are known to QEMU |
| 253 | (have corresponding "hv-*" flag) and copies "hv-spinlocks="/"hv-vendor-id=" |
| 254 | values from KVM to QEMU. "hv-passthrough" overrides all other "hv-*" settings on |
| 255 | the command line. Also, enabling this flag effectively prevents migration as the |
| 256 | list of enabled enlightenments may differ between target and destination hosts. |
Vitaly Kuznetsov | e48ddcc | 2019-05-17 16:19:20 +0200 | [diff] [blame] | 257 | |
Vitaly Kuznetsov | 70367f0 | 2021-09-02 11:35:26 +0200 | [diff] [blame] | 258 | 4.2. hv-enforce-cpuid |
| 259 | ===================== |
| 260 | By default, KVM allows the guest to use all currently supported Hyper-V |
| 261 | enlightenments when Hyper-V CPUID interface was exposed, regardless of if |
| 262 | some features were not announced in guest visible CPUIDs. 'hv-enforce-cpuid' |
| 263 | feature alters this behavior and only allows the guest to use exposed Hyper-V |
| 264 | enlightenments. |
Vitaly Kuznetsov | e48ddcc | 2019-05-17 16:19:20 +0200 | [diff] [blame] | 265 | |
Vitaly Kuznetsov | 70367f0 | 2021-09-02 11:35:26 +0200 | [diff] [blame] | 266 | |
| 267 | 5. Useful links |
Vitaly Kuznetsov | fb19f72 | 2019-05-17 16:19:19 +0200 | [diff] [blame] | 268 | ================ |
| 269 | Hyper-V Top Level Functional specification and other information: |
| 270 | https://github.com/MicrosoftDocs/Virtualization-Documentation |