Difference Engine:
Harnessing Memory Redundancy
in Virtual Machines
abstract
Virtual machine monitors (VMMs) are a popular platform
for Internet hosting centers and cloud-based compute services. By multiplexing hardware resources among virtual
machines (VMs) running commodity operating systems,
VMMs decrease both the capital outlay and management
overhead of hosting centers. Appropriate placement and
migration policies can take advantage of statistical multiplexing to effectively utilize available processors. However,
main memory is not amenable to such multiplexing and is
often the primary bottleneck in achieving higher degrees of
consolidation.
Previous efforts have shown that content-based page
sharing provides modest decreases in the memory footprint
of VMs running similar operating systems and applications. Our studies show that significant additional gains can
be had by leveraging both subpage level sharing (through
page patching) and incore memory compression. We build
Difference Engine, an extension to the Xen VMM, to support
each of these—in addition to standard copy-on-write full-page sharing—and demonstrate substantial savings across
VMs running disparate workloads (up to 65%). In head-to-head memory-savings comparisons, Difference Engine
outperforms VMware ESX server by a factor 1. 6–2. 5 for heterogeneous workloads. In all cases, the performance overhead of Difference Engine is less than 7%.
1. intRoDuction
Virtualization technology has improved dramatically over
the past decade to become pervasive within the service-delivery industry. Virtual machines are particularly attractive for server consolidation. Their strong resource and
fault isolation guarantees allow multiplexing of hardware among individual services, each with (potentially)
distinct software configurations. Anecdotally, individual server machines often run at 5–10% CPU utilization.
Operators’ reasons are manifold: because of the need to
over-provision for peak levels of demand, because fault
isolation mandates that individual services run on individual machines, and because many services often run
best on a particular operating system configuration. The
promise of virtual machine technology for server consolidation is to run multiple services on a single physical
machine while still allowing independent configuration
and failure isolation.
While physical CPUs are frequently amenable to
multiplexing, main memory is not. Many services run
comfortably on a machine with 1GB of RAM; multiplexing
10 VMs on that same host, however, would allocate each just
100MB of RAM. Increasing a machine’s physical memory is
often both difficult and expensive. Incremental upgrades
in memory capacity are subject to both the availability of
extra slots on the motherboard and the ability to support
higher-capacity modules: such upgrades often involve
replacing—as opposed to just adding—memory chips.
Moreover, not only is high-density memory expensive, it
also consumes significant power. Furthermore, as many-core processors become the norm, the bottleneck for VM
multiplexing will increasingly be the memory, not the
CPU. Finally, both applications and operating systems are
becoming more and more resource intensive over time. As
a result, commodity operating systems require significant
physical memory to avoid frequent paging.
Not surprisingly, researchers and commercial VM software vendors have developed techniques to decrease the
memory requirements for virtual machines. Notably, the
VMware ESX server implements content-based page sharing, which has been shown to reduce the memory footprint
of multiple, homogeneous virtual machines by 10–40%. 19
We find that these values depend greatly on the operating system and configuration of the guest VMs. We are
not aware of any previously published sharing figures for
mixed-OS ESX deployments. Our evaluation indicates, however, that the benefits of ESX-style page sharing decrease as
the heterogeneity of the guest VMs increases, due in large
part to the fact that page sharing requires the candidate
pages to be identical.
The premise of this work is that there are significant
additional benefits from sharing at a subpage granularity,
i.e., there are many pages that are nearly identical. We show
that it is possible to efficiently find such similar pages and
coalesce them into a much smaller memory footprint.
Among the set of similar pages, we are able to store most as
patches relative to a single baseline page. We also compress
The original version of this paper is entitled “Differ-
ence Engine: Harnessing Memory Redundancy in Vir-
tual Machines” and was presented at USENIX OSDI,
December 2008. An extended abstract entitled “Difference
Engine” appeared in USENIX ;login; volume 34, number 2.