figure 4. in most loosely coupled systems, messages may arrive multiple times and out
CB BA A
that it cannot process. Guaranteed
delivery means the messaging system
delivered it, but there is no guarantee
the message was well formed or, even
if it was, that the tempestuous application did something reasonable with the
message. Before my wife started doing
our household bills and it was my responsibility, the reliable delivery of the
electric bill to our house was only loosely correlated to the electric company receiving its money.
When considering the behavior of
the underlying message transport, it
is best to remember what is promised.
Each message is guaranteed to be delivered zero or more times! That is a guarantee you can count on. There is a lovely
probability spike showing that most
messages are delivered one time.
If you do not assume the underlying
transport may drop or repeat messages,
then you will have latent bugs in your
application. More interesting is the
question of how much help the plumbing layered on top of the transport can
give you. If the communicating applications run on top of plumbing that
shares common abstractions for messaging, some help may exist. In most
environments, the app must cope with
this issue by itself.
messaging system retries transmitting.
The messaging system often uses TCP,
which has its own mechanism to ensure the reliable delivery of bytes from
process to process. TCP’s guarantees
are real but apply only to a single process chatting with exactly one other
process. Challenges arise when longer-lived participants are involved.
Consider, for example, HTTP Web
requests. HTTP typically shuts down
the TCP connection between requests.
When a persistent HTTP connection is
used, the TCP connection is typically
left alive, but there is no guarantee.
This means any use of HTTP on top of
TCP may result in multiple sends of the
HTTP request. For this reason, most
HTTP requests are idempotent.
In scalable Web-service worlds, we
are constantly reimplementing the
same sliding window protocol2 that is
so ubiquitous in TCP, where the endpoints are running processes. The failure of either of the processes means
the failure of the TCP connection. In a
long-running messaging environment
implemented by a collection of servers,
the semantics of the endpoint are more
complex. As the representation of an
endpoint and its state evolves, so do the
messaging anomalies that are (
hopefully) managed by the plumbing. More
likely, they are incrementally solved by
the application as patches to surprising bugs. Either way, even application
developers will need a copy of Andrew
Tanenbaum’s classic book, Computer
Networks, at their fingertips.
˲Processing withdrawal ‘XYZ’ for
$1,000,000,000 if not already processed
˲ Baking a cake is not idempotent.
˲ Baking a cake starting from a shopping list (if you do not care about money) is idempotent.
˲ Reading record X is idempotent.
Even if the value changes, any legitimate value for X during the window between the issuance of the read and the
return of the answer is correct.
The definition of idempotent in
computer usage is: “Acting as if used
only once, even if used multiple times.”
While this is true, there are frequently
side effects of the multiple attempts.
Let’s consider a few side effects that are
not typically considered semantically
˲ Heaps. Imagine a system that uses
a heap during the processing of a request. You would naturally expect the
heap might become more fragmented
with multiple requests.
˲ Logging and monitoring. Most server
systems maintain logs that allow analysis and monitoring of the system. Repeated requests will influence the contents of the logs and the monitoring
˲ Performance. The repeated requests
may consume computation, network,
and/or storage resources. This may be
a tax on the throughput of the system.
These side effects are not relevant to
the semantics of the application behavior, so the processing of an idempotent
figure 5. When communicating with
service foo, multiple machines may
implement the service.
Why isn’t tCP enough?
TCP has had a major impact on unifying the ways in which we perform data
5 It offers exactly-once
and in-order byte delivery between two
communicating processes. It offers no
guarantees once the connection is terminated or one of the processes completes or fails. This means it covers only
a small portion of the landscape visible
to developers building reliable applications in a loosely coupled distributed
system. Realistically, the application
layers on top of TCP and must solve
many of the same problems all over
Requests get lost, so just about every
Foo-A Foo-B Foo-C Foo-N ...
To review, idempotence means that
multiple invocations of some work are
identical to exactly one invocation.
˲ Sweeping the floor is idempotent.
If you sweep it multiple times, you still
get a clean floor.
˲Withdrawing $1,000,000,000 is
not idempotent. Stutering and retrying
might be annoying.