The Xen VMM does not perform any I/O (delegating all
I/O to Domain-0) and is not aware of any devices. Thus, it
is not possible to build swap support directly in the VMM.
Further, since Difference Engine supports unmodified OSes,
we cannot expect any support from the guest OS. Hence, we
implement a single swap daemon (swapd) running as a user
process in Domain-0 to manage swap space. For each VM in
the system, swapd creates a separate thread to handle swap-in requests.
To swap out a page, swapd makes a hypercall into
Xen, where a victim page is chosen by invoking the global
clock. If the victim is a compressed or patched page, we
first reconstruct it. We pause the VM that owns the page
and copy the contents of the page to a page in Domain-0’s
address space (supplied by swapd). Next, we remove all
entries pointing to the victim page in the P2M and M2P
maps, and in the shadow page tables. We then mark the
page as swapped out in the corresponding page table entry.
Meanwhile, swapd writes the page contents to the swap
file and inserts the corresponding byte offset in a hash
table keyed by <Domain ID, guest page-frame number>.
Finally, we free the page, return it to the domain heap, and
reschedule the VM.
When a VM tries to access a swapped page, it incurs a
page fault and traps into Xen. We pause the VM and allocate
a fresh page to hold the swapped in data. We populate the
P2M and M2P maps appropriately to accommodate the new
page. Xen dispatches a swap-in request to swapd containing the domain ID and the faulting page-frame number. The
handler thread for the faulting domain in swapd receives
the request and fetches the location of the page in the swap
file from the hash table. It then copies the page contents
into the newly allocated page frame within Xen via another
hypercall. At this point, swapd notifies Xen, and Xen restarts
the VM at the faulting instruction.
We first present micro-benchmarks to evaluate the cost of
individual operations, the performance of the global clock
and the behavior of each of the three mechanisms in isolation. Next, we evaluate whole system performance: for a
range of workloads, we measure memory savings and the
impact on application performance. We present head-to-head comparisons with the VMware ESX server. Finally,
we demonstrate how our memory savings can be used to
boost the aggregate system performance. Unless otherwise mentioned, all experiments are run on dual-proces-sor, dual-core 2.33-GHz Intel Xeon machines and the page
size is 4KB.
5. 1. cost of individual operations
Before quantifying the memory savings provided by
Difference Engine, we measure the overhead of various
functions involved. Table 2 shows the overhead imposed by
the major Difference Engine operations. As expected, collapsing identical pages into a copy-on-write shared page
(share_page) and recreating private copies (cow_break)
are relatively cheap operations, taking approximately 6 and
25ms, respectively. Perhaps more surprising, however, is
90 communicationS of the acm | OcTOber2010 | VOL. 53 | nO. 10
table 2. cPu overhead of different functions.
function mean execution time (ms)
share_pages 6. 2
cow_break 25. 1
compress_page 29. 7
uncompress 10. 4
unpatch 18. 6
swap_out_page 48. 9
that compressing a page on our hardware is fast, requiring
slightly less than 30ms on average. Patching, on the other
hand, is almost an order of magnitude slower: creating a
patch (patch_page) takes over 300 ms. This time is primarily due to the overhead of finding a good candidate base page
and constructing the patch. Both decompressing a page and
reconstructing a patched page are also fairly fast, taking 10
and 18 ms respectively.
Swapping out takes approximately 50ms. However, this
does not include the time to actually write the page to disk.
This is intentional: once the page contents have been copied to user space, they are immediately available for being
swapped in; and the actual write to the disk might be delayed
because of file system and OS buffering in Domain-0.
Swapping in, on the other hand, is the most expensive operation, taking approximately 7ms. There are a few caveats,
however. First, swapping in is an asynchronous operation
and might be affected by several factors, including process
scheduling within Domain-0; it is not a tight bound. Second,
swapping in might require reading the page from disk, and
the seek time will depend on the size of the swap file, among
5. 2. Real-world applications
We now present the performance of Difference Engine on a
variety of workloads. We seek to answer two questions. First,
how effective are the memory-saving mechanisms at reducing memory usage for real-world applications? Second, what
is the impact of those memory-sharing mechanisms on
system performance? Since the degree of possible sharing
depends on the software configuration, we consider several
different cases of application mixes.
To put our numbers in perspective, we conduct head-to-head comparisons with VMware ESX server for three different workload mixes. We run ESX Server 3.0.1 build 32,039 on
a Dell PowerEdge 1950 system. Note that even though this
system has two 2.3-GHz Intel Xeon processors, our VMware
license limits our usage to a single CPU. We therefore restrict
Xen (and, hence, Difference Engine) to use a single CPU for
fairness. We also ensure that the OS images used with ESX
match those used with Xen, especially the file system and disk
layout. Note that we are only concerned with the effectiveness
of the memory sharing mechanisms—not in comparing the
application performance across the two hypervisors. In an
effort to compare the performance of Difference Engine to
ESX in its most aggressive configuration, we configure both
to scan 10,000 pages/s, the highest rate allowed by ESX.a