[Qemu-discuss] Supported hypervisors running VMs in nested VM

[Rain Maker]

Hello list,

I am trying to setup a lab environment for a backup solution. This lab
should be able to demonstrate the interaction the software has with
certain hypervisors, such as VMWare, Hyper-V and KVM, and the VMs
therein. My main focus is currently on Hyper-V.

I've enabled nested virtualization in the kernel module. When booting
a Linux-based LiveCD, I can see the vmx flag being passed to the VM.

However, I am unable to run any virtual machines within the hypervisors.

I've done tests on a couple of versions of Windows / Hyper-V, and
these are the results. Host system is Ubuntu 15.10, KVM version is
2.3.

Windows 2008 R2
-------------------------
Installation of Hyper-V via add / remove features works.
Booting works (which, because HV is a "type 1" hypervisor, is also a
special kind of Virtual Machine). However, I am unable to boot guest
VMs within this HV Instance.

The Event Viewer in Windows shows that 2 drivers are unable to load:
'Virtualization Infrastructure' driver required by the Virtual Machine
Management service is not installed or is disabled. Check your
settings or try reinstalling the Hyper-V role.
'Virtual Machine Bus' driver required by the Virtual Machine
Management service is not installed or is disabled. Check your
settings or try reinstalling the Hyper-V role.

Device Manager shows these 2 devices, with an error state of "Windows
cannot load the device driver for this hardware. The driver may be
corrupted or missing. (Code 39)"

Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".

This check can be skipped by using a different method of installation
(from PowerShell):
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart

This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.

Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.

I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.

Is this a problem specific to Hyper-V? Is there a method to get
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should
not be completely impossible (but I dislike VMWare for different
reasons).

My Qemu command line (generated by virt-manager). Except for disks and
domain names, all are identical:

qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on

Thank you in advance for response.

Sincerely,
Roel Brook

[Bandan Das]

...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.

Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Rain Maker]

Thanks Bandan.

That helped a bit. It got me to the next hurdle, as you suspected.

I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."

Apparently, Microsoft queries the BIOS to verify that the
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.

The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.

1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag

2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf

3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag

4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.

5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.

Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Bandan Das]

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.

Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.

Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Rain Maker]

Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.
As does VirtualBox FYI.

Booting with UEFI didn't make any difference.

After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For
Hyper-V to work (from what I understand), it should be either 4 or 5.

Googling that, funny enough, brought me back to this list:
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...

That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?

Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Bandan Das]

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Rain Maker]

Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.

But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.

However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
Loading Image...

I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.

Is this how the MSR is supposed to react?

AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)

Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Bandan Das]

Post by Rain Maker
Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.
But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.
However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.
Is this how the MSR is supposed to react?
AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)
Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

You have to use kvm to run (hardware assisted) nested virtualization. I am not
sure why you think Windows will read 0 for the feature control msr but you have to
use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2,
you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available
hardware features available to it and attempt to enable hardware virtualization.

Post by Rain Maker
Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Rain Maker]

The screenshot and weird behavior I posted is within the L1 VM.

So;
- L0 (host / hypervisor): 0x3a = 5 immediately after boot. L1 VM is
booted WITH -enable-kvm, nested=1, -hypervisor (also tried without
this option), +vmx

- L1 - Linux (VM / "sub-hypervisor"). 0x3a = 0 immediately after boot.
When a L2 VM is booted with -enable-kvm, 0x3a changes to 5 .
- L1 - Windows. 0x3a = ? (most likely, 0) Windows doesn't have tooling
to read MSR as far as I could find.

- L2 (under L1 Linux) - Boots fine. Doesn't matter whether I use
-enable-kvm or not
- L2 (under L1 Windows) - Does not start

As far as I understand it, the BIOS / UEFI should set that MSR to "5".
It should (again, as far as I understand, which is not that much) not
be the task of the operating system.

So, my question is;
- Why would the MSR 0x3a be 0 after boot?
- Why would it change to 5 after starting a L2 VM with -enable-kvm?
- Is it the responsibility of the BIOS / UEFI code to set that MSR (as
it does on my L0 host), or should the OS set this MSR appropriately?

It looks to me like this is a bug somewhere in the Qemu / KVM BIOS
code (MSR returned inappropriately). KVM seems to have a way to
automatically correct this, but Windows does not. I tried this on a VM
booted with the built-in seabios, as well as a VM using the OVMF UEFI
firmware. No difference in behavior.

Thank you very much for the help so far.

Roel Brook

Post by Bandan Das

Post by Rain Maker
Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.
But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.
However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.
Is this how the MSR is supposed to react?
AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)
Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

You have to use kvm to run (hardware assisted) nested virtualization. I am not
sure why you think Windows will read 0 for the feature control msr but you have to
use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2,
you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available
hardware features available to it and attempt to enable hardware virtualization.

Post by Rain Maker
Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Bandan Das]

Post by Rain Maker
The screenshot and weird behavior I posted is within the L1 VM.
So;
- L0 (host / hypervisor): 0x3a = 5 immediately after boot. L1 VM is
booted WITH -enable-kvm, nested=1, -hypervisor (also tried without
this option), +vmx
- L1 - Linux (VM / "sub-hypervisor"). 0x3a = 0 immediately after boot.
When a L2 VM is booted with -enable-kvm, 0x3a changes to 5 .
- L1 - Windows. 0x3a = ? (most likely, 0) Windows doesn't have tooling
to read MSR as far as I could find.
- L2 (under L1 Linux) - Boots fine. Doesn't matter whether I use
-enable-kvm or not
- L2 (under L1 Windows) - Does not start
As far as I understand it, the BIOS / UEFI should set that MSR to "5".
It should (again, as far as I understand, which is not that much) not
be the task of the operating system.
So, my question is;
- Why would the MSR 0x3a be 0 after boot?
- Why would it change to 5 after starting a L2 VM with -enable-kvm?
- Is it the responsibility of the BIOS / UEFI code to set that MSR (as
it does on my L0 host), or should the OS set this MSR appropriately?

L1 doesn't see the "real" feature control msr. It sees an emulated version.
From your experiment, it seems that kvm "sets" it up only after the
initial stages of running a guest - vmon/vmload etc. So, you get an
expected value only after you start running a guest.

Post by Rain Maker
It looks to me like this is a bug somewhere in the Qemu / KVM BIOS
code (MSR returned inappropriately). KVM seems to have a way to
automatically correct this, but Windows does not. I tried this on a VM
booted with the built-in seabios, as well as a VM using the OVMF UEFI
firmware. No difference in behavior.

This is KVM's responsibility and I believe that the correct behavior would
be set this if nested = 1 (if that is how it works on real hardware)
In that sense, it would be a bug but wouldn't be any useful.

Post by Rain Maker
Thank you very much for the help so far.

BTW you are mixing up cases by using/not using "-enable-kvm" in your command
line. What I mentioned is specific to kvm only, qemu probably will always
return a 0 for certain msrs (like this one).

Post by Rain Maker
Roel Brook

Post by Bandan Das

Post by Rain Maker
Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.
But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.
However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.
Is this how the MSR is supposed to react?
AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)
Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

You have to use kvm to run (hardware assisted) nested virtualization. I am not
sure why you think Windows will read 0 for the feature control msr but you have to
use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2,
you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available
hardware features available to it and attempt to enable hardware virtualization.

Post by Rain Maker
Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Rain Maker]

Thanks again Bandan.

One more question for you. In the KVM source tree (arch/x86/kvm) of
kernel 4.3-rc4, I see a file "hyperv.c". Comment states:

* KVM Microsoft Hyper-V emulation
*
* derived from arch/x86/kvm/x86.c
*
* Copyright (C) 2006 Qumranet, Inc.
* Copyright (C) 2008 Qumranet, Inc.
* Copyright IBM Corporation, 2008
* Copyright 2010 Red Hat, Inc. and/or its affiliates.
* Copyright (C) 2015 Andrey Smetanin <***@virtuozzo.com>
*

According to http://lxr.free-electrons.com/source/arch/x86/kvm/, this
file was not there in 4.2.

Do you know anything about this (seemingly recent) addition?

On my own laptop, I also made a bit of headway. I'm now at the point
where Hyper-V successfully installs. Did a brute hack to have
vmx_get_msr() return "5" each time MSR_IA32_FEATURE_CONTROL is called.
Not really the right way, but I'll worry about that once I actually
get it running.

I am still unable to run L2 VMs though. Getting errors like
"Hypervisor is not running". "systeminfo" now says everything is fine:

Hyper-V Requirements: VM Monitor Mode Extensions: Yes
Virtualization Enabled In Firmware: Yes
Second Level Address Translation: Yes
Data Execution Prevention Available: Yes

Which is why the installation now at least continues.

When starting a L2 VM, these MSR (decimal) seem to get called
(obtained by an equally hackish printk statement at the top of
vmx_get_msr()). I guess I'll go by them one by one and see which one
is "off". When booting the L1 VM, I also see a lot of negative numbers
in there.
Tried enabling ignore_msrs, but that didn't make any difference.

I'll report back when (if) I get it actually running, because I
believe it would really be cool if KVM were to support nested HV.

MSRs called when a L2 machine is started:
1025
1029
1033
1037
1041
1045
1049
1053
1057
1061

The top ones are actually all "MC0_STATUS". They return "0" on my host
machine, and do the same in L1 VMs. So it must be one of the other
ones. Unfortunately, most of them are not listed in the Intel spec..
http://www.cs.inf.ethz.ch/stricker/lab/doc/intel-part4.pdf

Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
The screenshot and weird behavior I posted is within the L1 VM.
So;
- L0 (host / hypervisor): 0x3a = 5 immediately after boot. L1 VM is
booted WITH -enable-kvm, nested=1, -hypervisor (also tried without
this option), +vmx
- L1 - Linux (VM / "sub-hypervisor"). 0x3a = 0 immediately after boot.
When a L2 VM is booted with -enable-kvm, 0x3a changes to 5 .
- L1 - Windows. 0x3a = ? (most likely, 0) Windows doesn't have tooling
to read MSR as far as I could find.
- L2 (under L1 Linux) - Boots fine. Doesn't matter whether I use
-enable-kvm or not
- L2 (under L1 Windows) - Does not start
As far as I understand it, the BIOS / UEFI should set that MSR to "5".
It should (again, as far as I understand, which is not that much) not
be the task of the operating system.
So, my question is;
- Why would the MSR 0x3a be 0 after boot?
- Why would it change to 5 after starting a L2 VM with -enable-kvm?
- Is it the responsibility of the BIOS / UEFI code to set that MSR (as
it does on my L0 host), or should the OS set this MSR appropriately?

L1 doesn't see the "real" feature control msr. It sees an emulated version.
From your experiment, it seems that kvm "sets" it up only after the
initial stages of running a guest - vmon/vmload etc. So, you get an
expected value only after you start running a guest.

Post by Rain Maker
It looks to me like this is a bug somewhere in the Qemu / KVM BIOS
code (MSR returned inappropriately). KVM seems to have a way to
automatically correct this, but Windows does not. I tried this on a VM
booted with the built-in seabios, as well as a VM using the OVMF UEFI
firmware. No difference in behavior.

This is KVM's responsibility and I believe that the correct behavior would
be set this if nested = 1 (if that is how it works on real hardware)
In that sense, it would be a bug but wouldn't be any useful.

Post by Rain Maker
Thank you very much for the help so far.

BTW you are mixing up cases by using/not using "-enable-kvm" in your command
line. What I mentioned is specific to kvm only, qemu probably will always
return a 0 for certain msrs (like this one).

Post by Rain Maker
Roel Brook

Post by Bandan Das

Post by Rain Maker
Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.
But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.
However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.
Is this how the MSR is supposed to react?
AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)
Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

You have to use kvm to run (hardware assisted) nested virtualization. I am not
sure why you think Windows will read 0 for the feature control msr but you have to
use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2,
you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available
hardware features available to it and attempt to enable hardware virtualization.

Post by Rain Maker
Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook

[Bandan Das]

Hi Roel,

Post by Rain Maker
Thanks again Bandan.
One more question for you. In the KVM source tree (arch/x86/kvm) of
* KVM Microsoft Hyper-V emulation
*
* derived from arch/x86/kvm/x86.c
*
* Copyright (C) 2006 Qumranet, Inc.
* Copyright (C) 2008 Qumranet, Inc.
* Copyright IBM Corporation, 2008
* Copyright 2010 Red Hat, Inc. and/or its affiliates.
*
According to http://lxr.free-electrons.com/source/arch/x86/kvm/, this
file was not there in 4.2.
Do you know anything about this (seemingly recent) addition?

As per my understanding, the Hyper V architecture has a paravirtualization
interface called Enlightenment and this file has the relevant msrs and the
associated logic. I took a quick look at the commit message, seems it has
just been moved.

Post by Rain Maker
On my own laptop, I also made a bit of headway. I'm now at the point
where Hyper-V successfully installs. Did a brute hack to have
vmx_get_msr() return "5" each time MSR_IA32_FEATURE_CONTROL is called.
Not really the right way, but I'll worry about that once I actually
get it running.

Ok, great. I think we should return what Hyper-V is expecting. Definitely
worth considering fixing if it helps in making progress with the installation.

Post by Rain Maker
I am still unable to run L2 VMs though. Getting errors like
Hyper-V Requirements: VM Monitor Mode Extensions: Yes
Virtualization Enabled In Firmware: Yes
Second Level Address Translation: Yes
Data Execution Prevention Available: Yes
Which is why the installation now at least continues.
When starting a L2 VM, these MSR (decimal) seem to get called
(obtained by an equally hackish printk statement at the top of
vmx_get_msr()). I guess I'll go by them one by one and see which one
is "off". When booting the L1 VM, I also see a lot of negative numbers
in there.
Tried enabling ignore_msrs, but that didn't make any difference.

Note that access to any unimplemented msrs should print out messages to the
kernel logs and dmesg should show them. Please also look for other relevant messages
in the L0 dmesg. This might give more clues as to what's missing.

Post by Rain Maker
I'll report back when (if) I get it actually running, because I
believe it would really be cool if KVM were to support nested HV.
1025
1029
1033
1037
1041
1045
1049
1053
1057
1061
The top ones are actually all "MC0_STATUS". They return "0" on my host
machine, and do the same in L1 VMs. So it must be one of the other
ones. Unfortunately, most of them are not listed in the Intel spec..

I am assuming you are running a Windows L2. So, what might be happening is
that the L2 guest might be trying to access the enlightenment msrs which ofcourse
traps to L0. But we might not be doing the right thing with them. (Just a guess)

Thank you for keeping us posted!

Post by Rain Maker
http://www.cs.inf.ethz.ch/stricker/lab/doc/intel-part4.pdf
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
The screenshot and weird behavior I posted is within the L1 VM.
So;
- L0 (host / hypervisor): 0x3a = 5 immediately after boot. L1 VM is
booted WITH -enable-kvm, nested=1, -hypervisor (also tried without
this option), +vmx
- L1 - Linux (VM / "sub-hypervisor"). 0x3a = 0 immediately after boot.
When a L2 VM is booted with -enable-kvm, 0x3a changes to 5 .
- L1 - Windows. 0x3a = ? (most likely, 0) Windows doesn't have tooling
to read MSR as far as I could find.
- L2 (under L1 Linux) - Boots fine. Doesn't matter whether I use
-enable-kvm or not
- L2 (under L1 Windows) - Does not start
As far as I understand it, the BIOS / UEFI should set that MSR to "5".
It should (again, as far as I understand, which is not that much) not
be the task of the operating system.
So, my question is;
- Why would the MSR 0x3a be 0 after boot?
- Why would it change to 5 after starting a L2 VM with -enable-kvm?
- Is it the responsibility of the BIOS / UEFI code to set that MSR (as
it does on my L0 host), or should the OS set this MSR appropriately?

L1 doesn't see the "real" feature control msr. It sees an emulated version.
From your experiment, it seems that kvm "sets" it up only after the
initial stages of running a guest - vmon/vmload etc. So, you get an
expected value only after you start running a guest.

Post by Rain Maker
It looks to me like this is a bug somewhere in the Qemu / KVM BIOS
code (MSR returned inappropriately). KVM seems to have a way to
automatically correct this, but Windows does not. I tried this on a VM
booted with the built-in seabios, as well as a VM using the OVMF UEFI
firmware. No difference in behavior.

This is KVM's responsibility and I believe that the correct behavior would
be set this if nested = 1 (if that is how it works on real hardware)
In that sense, it would be a bug but wouldn't be any useful.

Post by Rain Maker
Thank you very much for the help so far.

BTW you are mixing up cases by using/not using "-enable-kvm" in your command
line. What I mentioned is specific to kvm only, qemu probably will always
return a 0 for certain msrs (like this one).

Post by Rain Maker
Roel Brook

Post by Bandan Das

Post by Rain Maker
Unfortunately, no difference. WIth or without -hypervisor doesn't make
any difference to that flag.
But, experimenting on, I found something <very> odd.
The 0x3a register is 0 when the VM boots up.
Even when I start a L2 VM, 0x3a is still 0.
However, once I start WITH -enable-kvm, 0x3a is suddenly 5(!). See
this terminal session, which is executed within the L1 VM ("kvmtest").
http://storage4.static.itmages.com/i/15/1006/h_1444163503_6656916_6ffbfd2352.png
I was only executing mini.iso (an Ubuntu Netinstaller), and closing it
at the boot prompt. I did not do anything in the L2 VM. Both the Qemu
VM as the -enable-kvm VMs do boot.
Is this how the MSR is supposed to react?
AFAIK, the MSR can only be modified from kernelspace (which also
explains why Qemu would only reset it with -enable-kvm, there are no
kernelspace components used without it if I understand correctly)
Looking at this, I can imagine that Windows does not detect a correct
value. It will get 0. Would it make sense to cygwin KVM and see if
that changes the MSR register?

You have to use kvm to run (hardware assisted) nested virtualization. I am not
sure why you think Windows will read 0 for the feature control msr but you have to
use —enable-kvm in L0 when you are launching L1 (Hyper-V in your case). When L1 runs L2,
you don’t have to worry about using —enable-kvm. Hyper-V should automatically detect available
hardware features available to it and attempt to enable hardware virtualization.

Post by Rain Maker
Sincerely,
Roel Brook.

Post by Bandan Das

Post by Rain Maker
Qemu on Linux works fine. I did not even have to explicitly set
-hypervisor. It simply works.

Sorry, I meant running Linux with "-hypervisor" to see if specifying
that is somehow messing with the feature flags.

Post by Rain Maker
As does VirtualBox FYI.
Booting with UEFI didn't make any difference.
After A LOT of Googling, I believe that Hyper-V actually checks bit
0x3a of the MSR register (instead of, as the error would .
This is a 3 bit register (IA32_FEATURE_CONTROL). Within my VM, the
value returned by rdmsr is "0", while on my host it is "5". For

That seems odd. Even Linux wouldn't work if that value is 0.

Post by Rain Maker
Hyper-V to work (from what I understand), it should be either 4 or 5.
https://lists.gnu.org/archive/html/qemu-devel/2015-01/msg01371.html.
I guess it really IS important to know what you're googling for to
find things fast...
That thread simply says that the kernel is too old. Well, my host is
running 4.2, so should be new enough.
I'm a bit stuck. Any ideas?
Sincerely,
Roel Brook

Post by Bandan Das

Post by Rain Maker
Thanks Bandan.
That helped a bit. It got me to the next hurdle, as you suspected.
I modified the virsh XML so that -cpu host,+vmx,-hypervisor is passed,
and the installation now reports "Hyper-V cannot be installed because
virtualization support is not enabled in the BIOS.".

Thanks for trying this out.

Post by Rain Maker
I am sure that vmx is passed. but "systeminfo" does report "Hyper-V
cannot be installed because virtualization support is not enabled in
the BIOS."
Apparently, Microsoft queries the BIOS to verify that the

When kvm is initialized, it checks for TXT and VMX both being enabled.
That too, only if the feature control msr is locked. I don't think there
are actually any specific "bios calls" to find this out. I would assume
Hyper-V should be doing the same thing but your testing says otherwise.
Can you please run linux as L1 with "-hypervisor" and see if it works ?
If it doesn't, please check dmesg for relevant messages.
Thanks,
Bandan

Post by Rain Maker
virtualization bit is actually enabled, instead of simply relying on
the VMX flag.
Unfortunately, VMs are still not starting either. The seabios in Qemu
seems to be pretty difficult to modify. I'll check whether I can
reinstall on UEFI, maybe that is going to make a difference.
The way VMWare does this is actually semi-documented (it hasn't always
been in the product, and a workaround involving manually editing the
configuration has been used for a long time). I'll see if I can
correlate these to Qemu options, to see whether we can use those
instructions to get this working on Qemu.
1. Set 'vhv.enable = "TRUE" on the VM
It "enables virtual hardware virtualization". This seems equivalent
to the -hypervisor flag
2. Set 'monitor.virtual_exec = "hardware" on the VM.
This option seems to force hardware virtualization for both CPU and
MMU. Unsure whether there's an equivalent Qemu configuration option.
Unsure whether it's needed on Qemu. Details at
http://www.vmware.com/files/pdf/perf-vsphere-monitor_modes.pdf
3. Set hypervisor.cpuid.v0 = “FALSE” in the VM configuration
This seems synonymous to the +vmx flag
4. Enable the option to "Virtualize VT-x/EPT or AMD/RVI"
I have not found any option to explicitly do this in Qemu. Looking
at my Ubuntu VM, the "ept" flag IS passed to the VM, so this should be
OK.
5. Add the following CPU mask Level ECX: ---- ---- ---- ---- ---- ---- --H- ----
Not sure how to do that in Qemu or what it does. Looking at
https://en.wikipedia.org/wiki/CPUID, it seems to disable the XSAVE
instruction(?). For fun, I passed -cpu ...-xsave, but it did not seem
to make any difference whatsoever.
Sincerely,
Roel Brook

Post by Bandan Das
...

Post by Rain Maker
Windows 2012 / 2016 technical preview 3
--------------------------------------------------------
The installation via the "default" method of Add/Remove Features does
not work. Hyper-V displays the error message "A hypervisor is already
running".
This check can be skipped by using a different method of installation
Enable-WindowsOptionalFeature –Online -FeatureName Microsoft-Hyper-V
–All -NoRestart
This results in (again) the server booting up, but being unable to run
any guest VMs. The error message is less clear then that in 2008, just
"The Virtual Machine Management Service failed to start the virtual
machine 'New Virtual Machine' because one of the Hyper-V components is
not running (Virtual machine ID
0C063B29-249A-41C8-8A5B-6D4D2E37EF7C)."
is what I could find.
Other
--------
Just to verify that "nesting" is actually working, I've also installed
a Ubuntu 15.10 VM and installed Qemu on it.
This combination CAN successfully run a VM.
I've also installed VirtualBox on one of the Windows VMs. This
VirtualBox instance is also capable of running virtual machines.
According to the icon in the bottom right, VirtualBox IS using the
hardware virtualization.
Is this a problem specific to Hyper-V? Is there a method to get

Nesting a Hyper-V L1 hypervisor is largely untested. But one of the problems I recollect is that Hyper-V doesn’t like running in a virtualized environment. It checks the “hypervisor” feature flag that Qemu exports. You could try running qemu with “-cpu host,-hypervisor” or something similar and see if it makes any difference. I suspect there would be other roadblocks though, this is just one of them.

Post by Rain Maker
Hyper-V working including running guests? I know for a fact that
VMWare Workstation / ESX is able to run Hyper-V fully, so it should

Yes, IIRC one of the things ESX does is hide the hypervisor flag specifically for Hyper-V.
Bandan

Post by Rain Maker
not be completely impossible (but I dislike VMWare for different
reasons).
My Qemu command line (generated by virt-manager). Except for disks and
qemu-system-x86_64 -enable-kvm -name Windows_2008_R2 -S -machine
pc-i440fx-vivid,accel=kvm,usb=off -cpu
SandyBridge,+invtsc,+osxsave,+pcid,+pdcm,+xtpr,+tm2,+est,+smx,+vmx,+ds_cpl,+monitor,+dtes64,+pbe,+tm,+ht,+ss,+acpi,+ds,+vme
-m 2048 -realtime mlock=off -smp 2,sockets=2,cores=1,threads=1 -uuid
54a8f3a3-66c2-45a5-a280-ecf7019a67fa -no-user-config -nodefaults
-chardev socket,id=charmonitor,path=/var/lib/libvirt/qemu/Windows_2008_R2.monitor,server,nowait
-mon chardev=charmonitor,id=monitor,mode=control -rtc
base=localtime,driftfix=slew -global kvm-pit.lost_tick_policy=discard
-no-hpet -no-shutdown -global PIIX4_PM.disable_s3=1 -global
PIIX4_PM.disable_s4=1 -boot strict=on -device
ich9-usb-ehci1,id=usb,bus=pci.0,addr=0x6.0x7 -device
ich9-usb-uhci1,masterbus=usb.0,firstport=0,bus=pci.0,multifunction=on,addr=0x6
-device ich9-usb-uhci2,masterbus=usb.0,firstport=2,bus=pci.0,addr=0x6.0x1
-device ich9-usb-uhci3,masterbus=usb.0,firstport=4,bus=pci.0,addr=0x6.0x2
-device virtio-serial-pci,id=virtio-serial0,bus=pci.0,addr=0x5 -drive
file=/sub/kvm/Windows_2008_R2.qcow2,if=none,id=drive-ide0-0-0,format=qcow2,cache=unsafe,aio=threads
-device ide-hd,bus=ide.0,unit=0,drive=drive-ide0-0-0,id=ide0-0-0,bootindex=1
-drive file=/sub/ISO/en_windows_server_2008_r2_with_sp1_x64_dvd_617601.iso,if=none,id=drive-ide0-0-1,readonly=on,format=raw
-device ide-cd,bus=ide.0,unit=1,drive=drive-ide0-0-1,id=ide0-0-1
-netdev tap,fd=24,id=hostnet0 -device
rtl8139,netdev=hostnet0,id=net0,mac=52:54:00:7b:d7:d2,bus=pci.0,addr=0x3
-chardev pty,id=charserial0 -device
isa-serial,chardev=charserial0,id=serial0 -chardev
spicevmc,id=charchannel0,name=vdagent -device
virtserialport,bus=virtio-serial0.0,nr=1,chardev=charchannel0,id=channel0,name=com.redhat.spice.0
-device usb-tablet,id=input0 -spice
port=5903,addr=127.0.0.1,disable-ticketing,seamless-migration=on
-device qxl-vga,id=video0,ram_size=67108864,vram_size=67108864,vgamem_mb=16,bus=pci.0,addr=0x2
-device intel-hda,id=sound0,bus=pci.0,addr=0x4 -device
hda-duplex,id=sound0-codec0,bus=sound0.0,cad=0 -chardev
spicevmc,id=charredir0,name=usbredir -device
usb-redir,chardev=charredir0,id=redir0 -chardev
spicevmc,id=charredir1,name=usbredir -device
usb-redir,chardev=charredir1,id=redir1 -chardev
spicevmc,id=charredir2,name=usbredir -device
usb-redir,chardev=charredir2,id=redir2 -chardev
spicevmc,id=charredir3,name=usbredir -device
usb-redir,chardev=charredir3,id=redir3 -device
virtio-balloon-pci,id=balloon0,bus=pci.0,addr=0x7 -msg timestamp=on
Thank you in advance for response.
Sincerely,
Roel Brook