Enabling Highly Scalable Remote
Memory Access Programming
with MPI- 3 One Sided
By Robert Gerstenberger,* Maciej Besta, and Torsten Hoefler
Modern high-performance networks offer remote direct
memory access (RDMA) that exposes a process’ virtual
address space to other processes in the network. The
Message Passing Interface (MPI) specification has recently
been extended with a programming interface called MPI- 3
Remote Memory Access (MPI- 3 RMA) for efficiently exploiting state-of-the-art RDMA features. MPI- 3 RMA enables a
powerful programming model that alleviates many message
passing downsides. In this work, we design and develop
bufferless protocols that demonstrate how to implement
this interface and support scaling to millions of cores with
negligible memory consumption while providing highest
performance and minimal overheads. To arm programmers, we provide a spectrum of performance models for
RMA functions that enable rigorous mathematical analysis of application performance and facilitate the development of codes that solve given tasks within specified time
and energy budgets. We validate the usability of our library
and models with several application studies with up to half a
million processes. In a wider sense, our work illustrates how
to use RMA principles to accelerate computation- and data-intensive codes.
Supercomputers have driven the progress of various society’s domains by solving challenging and computationally intensive problems in fields such as climate modeling,
weather prediction, engineering, or computational physics.
More recently, the emergence of the “Big Data” problems
resulted in the increasing focus on designing high-performance architectures that are able to process enormous
amounts of data in domains such as personalized medicine, computational biology, graph analytics, and data
mining in general. For example, the recently established
Graph500 list ranks supercomputers based on their ability
to traverse enormous graphs; the results from November
2014 illustrate that the most efficient machines can process up to 23 trillion edges per second in graphs with more
than 2 trillion vertices.
Supercomputers consist of massively parallel nodes,
each supporting up to hundreds of hardware threads in a
single shared-memory domain. Up to tens of thousands
of such nodes can be connected with a high-performance
network, providing large-scale distributed-memory parallelism. For example, the Blue Waters machine has >700,000
cores and a peak computational bandwidth of > 13 petaflops.
Programming such large distributed computers is far
from trivial: an ideal programming model should tame the
complexity of the underlying hardware and offer an easy
abstraction for the programmer to facilitate the development of high-performance codes. Yet, it should also be able
to effectively utilize the available massive parallelism and
various heterogeneous processing units to ensure highest
scalability and speedups. Moreover, there has been a growing need for the support for performance modeling: a rigorous
mathematical analysis of application performance. Such
formal reasoning facilitates developing codes that solve
given tasks within the assumed time and energy budget.
The Message Passing Interface (MPI) 11 is the de facto standard API used to develop applications for distributed-memory
supercomputers. MPI specifies message passing as well as
remote memory access semantics and offers a rich set of features that facilitate developing highly scalable and portable
codes; message passing has been the prevalent model so far.
MPI’s message passing specification does not prescribe specific ways how to exchange messages and thus enables flexibility in the choice of algorithms and protocols. Specifically,
to exchange messages, senders and receivers may use eager
or rendezvous protocols. In the former, the sender sends a
message without coordinating with the receiver; unexpected
messages are typically buffered. In the latter, the sender waits
until the receiver specifies the target buffer; this may require
additional control messages for synchronization.
Despite its popularity, message passing often introduces
time and energy overheads caused by the rendezvous control
messages or copying of eager buffers; eager messaging may
also require additional space at the receiver. Finally, the
fundamental feature of message passing is that it couples
communication and synchronization: a message both trans-
fers the data and synchronizes the receiver with the sender.
This may prevent effective overlap of computation and
The original version of this paper was published in the
Proceedings of the Supercomputing Conference 2013 (SC’ 13),
Nov. 2013, ACM.
* RG performed much of the implementation during an internship at UIUC/
NCSA while the analysis and documentation was performed during a scientific
visit at ETH Zurich. RG’s primary email address is gerstenberger.robert@