VMMs Hide OS Limitations Instead of Fixing Them

April 4, 2015

Virtual machine monitors have become the default answer to multi-tenancy, isolation, and resource management not because they are the best solution, but because they mask the limitations of the operating systems underneath. Our ever-increasing reliance on VMMs is sub-optimal, and it is making us complacent about fixing the root problems.

The pattern repeats across decades. In the 1990s, when huge SMP systems first appeared, operating systems could not scale across many processors so VMMs were used to run multiple OS instances to utilize all those CPUs. Today, OSes handle SMP fine, but now we use VMMs to compensate for their poor multi-tenancy, resource isolation, and IO QoS guarantees. In Azure, we ran each web/worker role instance inside a separate VM even though it was essentially a single process, because the underlying OS could not provide adequate IO quota management and isolation at the kernel level.

The comparison between VMMs and microkernels is instructive but misleading. VMMs sidestep the hard problems rather than solving them. Xen avoids the page eviction problem by simply not oversubscribing RAM an exokernel could achieve the same. VMMs claim to reduce IPC overhead, but they often replace it with more expensive full-blown TCP between VMs on the same box, or require special hardware like SR-IOV to work around their own IPC limitations. Each VM runs a full OS with hundreds of megabytes of overhead, mostly duplicate copies of the same thing with no sharing through copy-on-write a grossly missed opportunity. And guest enlightenment is always required for decent performance, undermining the "run unmodified OSes" promise.

The real cost is in what does not get built. Rings 1-2 on Intel hardware remain completely underutilized. True kernel-level resource isolation is hard, but it is not impossible we simply have not invested in it because VMMs provide a good-enough workaround. Every year that passes with VMMs as the default is another year where OS isolation, QoS, and multi-tenancy do not improve.

Takeaway: VMMs are a patch, not a fix and the longer we rely on them, the less incentive there is to solve the underlying OS problems that made them necessary.