[feature-request] Nested Virtualization: VT-in-VT

[Technologov]

Hi All,

Because more and more virtualization software requires VT (Intel VMX or AMD-V) to run, it makes sense to virtualize it somehow. (at least if host CPU supports hardware VT)

It will allow the following software to run in guests: -Xen (full virtualization mode) -KVM -Hyper-V -Windows Virtual PC -- this is basically a requirement for those who wanna run Windows 7 "XP mode". i.e. It allows for more complete user experience on Win7 guests.

Others can take advantage of it too: -VirtualBox itself :) -VMware Workstation -VMware ESX

-Technologov (yes, I will take advantage of it too :-), yay! )

[Edit by klaus 2018-05-06:]

This feature is being worked on for a while. Once the implementation is complete and tested it will be available to all users.

[Technologov]

opened on 17.5.2009.

[Sander van Leeuwen]

A lot of work for questionable usefulness. Definitely very low on our priority list.

[Technologov]

This is partially related to wish #2988

-Technologov

[Technologov]

KVM team has developed (untested) patches for both nested VMX and nested SVM. Maybe some ideas can be taken from those patches.

See: (General Info)

​http://avikivity.blogspot.com/2008/09/nested-svm-virtualization-for-kvm.html

and

​http://avikivity.blogspot.com/2009/09/nested-vmx-support-coming-to-kvm.html

-Technologov

[Technologov]

update: VMware Workstation 8 claims to support this feature !

​https://www.vmware.com/support/ws80/doc/releasenotes_workstation_80.html

-Technologov

[Technologov]

It seems that Linux 3.1 KVM finally achieved nested VMX.

This feature is yet another filler feature that is a "must-have":

​http://www.mjmwired.net/kernel/Documentation/virtual/kvm/nested-vmx.txt

Research paper:

​http://www.usenix.org/events/osdi10/tech/full_papers/Ben-Yehuda.pdf

-Technologov

[Frank Mehnert]

What you denote as "must have" needs a lot of implementation time, is very difficult to implement and the benefit of nested virtualization is questionable.

[Frank Mehnert]

I know that other virtualization products support nested virtualization. Even with VirtualBox it is possible to execute VirtualBox without VT-x/AMD-V as a guest of VirtualBox with VT-x/AMD-V enabled. But apart from that, implementing full virtualization is too much effort for too little benefit.

[itarchitectlev]

It seems that this ticket was evaluated a long time ago when the usefulness may have been disputed. In any case, there are real reasons people want to have nested hardware VT.

There is a lot of working happening in the cloud community and evaluating products in this arena often requires hardware VT - which implies two things:

1) You have real hardware available 2) You have nested hardware VT capable virtualization software (e.g. VMware)

Given that we are in a world of cheap memory coupled with multiple core fast CPUs, virtualization is a great boost to doing cloud infrastructure development work. The problem is that the virtualized hardware created under VBox is only one part of the picture - as you're creating virtual hardware to run infrastructure designed to virtualize. Nested Hardware VT is essential here as software emulation in this world is painfully slow.

I have just purchased a VMware product because VirtualBox doesn't support this fundamental feature that I require which will now render my use of VirtualBox redundant.

I have been a big advocate of VirtualBox, but future work now sees this being left behind.

Regards, Kev @itarchitectkev

[Frank Mehnert]

I don't want to start a discussion here, a public bugtracker is not the right place for doing this. But please be aware that the VirtualBox development is primary driven by paying customer demands. Of course that does not mean that community demands are not being considered. We just have limited resources. Code contribution from the community may speed up the implementation of certain features.

[Senthil Nathan]

Will nested VM work using "Intel Haswell CPU" which has VMCS feature allowing nested virtualization. If it does not it makes sense (in my opinion) to add support for nested VM with New CPU VMCS feature rather than writing the emulation layer.

[Ramshankar Venkataraman]

No. Haswell CPUs while it has features like VMCS shadowing that help make Nested Virtualization perform better and easier to program, we still have to implement code which makes use of it. At the moment, we still feel when we implement the nested virtualization feature, having a Haswell CPU as the minimum requirement is not reasonable as it's still too new, and will exclude quite a bit of the users even two years down the line.

This feature is most definitely on our radar for a while now, but again, it's a question of managing our development resources which are limited.

[Michael Thayer]

I have removed most of the comments from this ticket, as they were more of a discussion nature and not very conductive to implementation. I added the nicknames of the commenters to the CC list so that they still get updates to the ticket. Unfortunately trac does not like nicknames with spaces in in CC, so I just removed the content of those comments.

As mentioned previously, we currently do not have the available resources to implement this feature. If people in the community are willing and able to help we would certainly be interested in code contributions, but in that case please communicate with us before you start writing code to avoid wasted effort. If anyone in the community would be willing to be hired to work on this several people have indicated that they would be interested in trying to organise funding. Alternatively perhaps someone can find a suitable freelance developer. The same thing applies here: please talk to us before starting writing.

[Pro Tipper]

cc me too please?

[Michael Thayer]

Pro Tipper: unfortunately the CC field can't handle user names with spaces in them. I will just leave your comment and that will cause you to get update notifications.

Current CC list is: Tao1, vbfun, DNS, Tristan, x5560, Tsso, bluezeak, peterdk, snowch, rdx, Bertrualex, EnesKorukcu, AfUnix, dario, anrichter, Jimbob, adam84luong, Coffee_fan, everflux, purpletoad, jonseymour, ciekawy, GA, ThatOneGuySean, Youness, subdian, veganaiZe, JohnStarich, Michal, todd.vierling, emiliano.bonassi, killmenot, ccarbone, azurtree, amclain, macedemo, GnomeUser, nicorac, schmunk, gael.abadin, paha, krasimir, pixie, Parfait, ikb42, yanp.bugz, hekier, maniankara, RayN, oam, domiel, sab3awy, Canada00, Ashishkel, leonexis, andjohn2000, denka_b, theBruno

[Gerry R]

cc me please

[Parkeren Schiphol]

Would really benefit from this for all my employees ​https://parkos.nl/parkeren-schiphol we are all running w7 in xp mode due to a software compatibility issue. This should be a basic feature.

[argenstijn]

Could you increase priority! Nowadays this feature will come in handy.

[einar]

cc me please

[jwatilo]

Just want to add my voice -- we NEED nested virtualization. With the latest technologies this is becoming a MUST.

[Socratis]

1. You don't "need" or "must have" nested virtualization, the vast majority of the world is doing great without it. You "would like" is more like it... ;)
2. Take a look at the timeline.
3. As Yoda once said:

   Patience you must have my young Padawan...

[mironex]

Replying to socratis:

1. You don't "need" or "must have" nested virtualization, the vast majority of the world is doing great without it. You "would like" is more like it... ;)
2. Take a look at the timeline.
3. As Yoda once said:

   Patience you must have my young Padawan...

Dear Socratis, Padwan is right. It's no longer a caprice. But an obvious need. Many tools, for example, for containerization are launched in a virtualbox.

That means that we really need to run a virtualbox in a virtualbox.

Maybe is possible to change the priorities. If you, Socratis, are so influential, you can change something.

If you do not have such an influence, well we must practice a Greek stoic patience or just use maybe e.g. KVM, VMWare.

Well, 9 years is a long time

Error: On virtual box is not able to start minikube

Starting local Kubernetes v1.9.0 cluster...
Starting VM...
E0214 22:14:50.003571    7064 start.go:159] Error starting host: Error creating host: Error executing step: Running precreate checks.
: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory.

 Retrying.
E0214 22:14:50.004044    7064 start.go:165] Error starting host:  Error creating host: Error executing step: Running precreate checks.
: This computer doesn't have VT-X/AMD-v enabled. Enabling it in the BIOS is mandatory

[mironex]

cc me too please?

[Socratis]

@mironex
You didn't take a look at the timeline, did you?

[n00ris]

cc me please

[n00ris]

I opened the timeline, what in special shloud we see at the timeline?

I used my browser search to find "nested" in the page and I found

Changeset in vbox [71004] by vboxsync

VMM/HMSVMR0: Nested Hw.virt: Fix intercepting VMMCALL while executing the …

from 2018-02-14

Does this mean this feature is Work In Progress, but not yet in the latest release? Is there a beta available, or is it in a state that we could test if I build virtualbox directly from svn?

[maeni70]

I don't understand the timeline either. And i agree that this feature is important! E.g. Android Emulator needs VT-X. I wanted to use VirtualBox VM for mobile development, but as I learned just right know, it is limited for this purpose.

[maeni70]

cc me too please. thanks

[Socratis]

The timeline simply tells you that fixes/patches are coming in all the time. It means that they're working on this feature. Hard. No, it's not ready, no, you didn't see any announcements. And keep in mind that yes, it may becoming an important feature, but a couple of years ago you didn't even have virtualization at all, so jumping from nothing to nested, that's quite a leap.

It will become available. Just (like everything else in life), don't ask for promises or specific dates. The source code is available, if you can't contribute, at least you can monitor it to see if they're working on it. And they are...

[paleozogt]

+1 cc me also

[jotenko]

I'm also waiting desperately for this. CC me too, please.

[vbkamfung]

As docker is becoming common, this request should be changed from 'want to' to 'must have'.

[Michael Thayer]

Based on the number of code contributions we have seen, or the number of people expressing interest in contributing (I think none at all, but I may have missed one or two) this feature cannot be in very high demand. Be that as it may, it is being worked on.

[Darren Mackay]

Just want to add my support for this.

This past week - I now have 3 clients (2x UK and 1 x NL) who are deploying Windows Containers as standard - 1 of these clients has 3000 users (also a large Oracle JDE house). the requirement to run windows as a VM is mandatory - due to lockdown by the customer requirements - and thus containers within the locked down windows image (also includes VPM clients / etc).

Note - Running virtualbox on Mac as a consultant.

[brexit]

I occasionally need to run KVM and in order to do that I must stop VBox VMs, disable VBox, enable KVM, reboot, and revert 1-2 days later. And of course migrate VBox VMs to KVM, if I need any of them on KVM.

Test/dev, containers, compatibility requirement (for KVM-specific ISVs), etc - there are many reasons why it is desirable.

[Socratis]

@brexit

Your first requirement is not about nested virtualization, it's about concurrent virtualization. And unfortunately, KVM (just like Hyper-V) doesn't want to play nice.

The trick with VT-x is that it has to be shared, since it's "only one" available slot for its use. VMware and VirtualBox do not use VT-x at the same time but rather take turns. Simply put, programs that use VT-x should do this: VT-x lock, execute, VT-x unlock. This way other processes can use VT-x in turns.

On the other hand, KVM and Hyper-V permanently take over VT-x and don't let anyone else use it. If a program uses VT-x from the moment it is loaded and doesn't let go till the moment it dies, you have a problem. So, please complain to the KVM and Hyper-V people about that.

[Stef2]

Raising the count for VT-in-VT feature. Main use case is also docker.