Applications are no longer islands. Not
only do they frequently run distributed and replicated over many cloud-based computers, but they also run
over many handheld computers. This
makes it challenging to talk about a
single truth at a single place or time.
In addition, most modern applications interact with other applications.
These interactions settle out to impact
understanding. Over time, a shared
opinion emerges just as new interactions add increasing uncertainty. Many
business, personal, and computational
“facts” are, in fact, uncertain. As some
changes settle, others meander from
place to place.
With all the regular, irregular, and
uncleared checks, my understanding
of our personal joint checking account
is a bit hazy. While I try to convince myself I will someday understand it, I have
reconciled myself that it’s really consistently eventual.
Don’t Settle for Eventual Consistency
Wyatt Lloyd et al.
Not What You Were Expecting?
Wojciech Golab et al.
1. Bailis, P. and Ghodsi, A. Eventual consistency today:
limitations, extensions, and beyond. acmqueue 11, 3
2. Helland, P. Data on the outside versus data on the
inside. In Proceedings of the Conference on Innovative
Database Research, 2005.
3. Helland, P. and Campbell, D. Building on quicksand.
In Proceedings of the Conference on Innovative
Database Research, 2009.
4. Mohan, C. and Lindsay, B. Efficient commit protocols
for the tree of processes model of distributed
transactions. ACM SIGOPS Operating Systems Review
19, 2 (1985), 40–52; https://dl.acm.org/citation.
5. Shapiro, M., Preguiça, N., Baquero, C. and Zawirski, M.
Conflict-free replicated data types. In Proceedings
of the 13th International Conference on Stabilization,
Safety, and the Security of Distributed Systems, 2011,
Springer-Verlag, Berlin, 386–400.
6. Vogels, W. Eventually consistent. Commun. ACM 52,
1 (Jan. 2009), 40–44; https://dl.acm.org/citation.
Pat Helland has been implementing transaction systems,
databases, application platforms, distributed systems,
fault-tolerant systems, and messaging systems since
since 1978. He currently works at Salesforce.
Copyright held by owner/author.
Publication rights licensed to ACM. $15.00.
There is also the pesky problem
of the check written to someone who
doesn’t deposit it in a reasonable
amount of time. Perhaps it was left in a
wallet and deposited a year later. If the
check is not deposited for months, do
you put a stop order on it or just wait
and assume it’s not coming through?
The balance in your checking account
is annoyingly eventual!
Snapping Uncertainty into
“What We Know So Far”
Our bank sends us monthly statements. They represent the debits and
credits to our account that cleared a
strongly consistent location as of a
deadline. That strongly consistent location is the bank’s centralized computer system. The debits and credits
that have arrived at the clearinghouse
by the monthly deadline get scooped
up into the account’s statement.
Quarterly reports for public companies take a similar but different approach. At midnight of the last day of
the quarter, new business and new expenses start being allotted to the next
quarter. The company begins gathering and organizing all the income
and expenses from the now-closed
quarter. Then all records of what was
spent and earned are swept into a big
mélange that results in a public quarterly profit and loss report. This usually happens within 40 days or so after
the quarter closes.
Some transactions during the quarter, however, may not be sent to the
accountants in a timely fashion. One
contributing factor may be the employment of software engineers, who are
notoriously bad at the punctual submission of expense reports. So, the results published for the quarter are approximately correct but not perfect.
After the quarterly report, corrections will dribble in to the accountants.
They will either categorize them as minor and issue a slight correction to the
numbers for the previous quarter or issue a restatement. For a public company, a restatement showing a noticeable
difference from the published report is
embarrassing and rarely happens. Minor corrections are common.
You can’t really know what happened
until you have heard everything.
The longer you wait, the more you
hear and the more accurate your opin-
ion. Eventually, you give up waiting for
new information and declare your opin-
ion of what happened a few months ago
is accurate enough.
In the bank account statement, the
definition of certainty is provided at
the bank’s centralized computer system at end-of-day when the month
closes. Uncertainty from the perspective
of the bank is eliminated. In the corporate quarterly reports, uncertainty is
gauged by how much of the underlying
truth of the business filters its way back
to the accountants. The quarterly report is not definitive, just pretty darned
close—at least usually.
Trust, Timeouts, and Escalation
Working across trust boundaries is
always eventual. Because you may
not trust another entity, you are not
going to do a distributed two-phase
commit4 and lock up your database
waiting for that other company. Instead, you have a workflow in which
partial trust is used to get your cooperative business done. Throughout
this process, there are long windows
in which you are just waiting and waiting … still more examples of being
This eventual nature of uncertainty
continues through the steps of the
workflow. You agree to reserve 200
widgets from your inventory on the receipt of a deposit from your purchaser.
If you hear nothing back to consummate the purchase of the widgets, you
are stuck. While your disappointment
is somewhat tempered by the deposit
you keep, it’s not enough to pay for the
widgets. Darn! You lost out on selling
them to another customer!
Working across trust boundaries,
cooperative work functions using
timeouts and escalation. If you do
not call to cancel your room reservation at a hotel with 72 hours advance
notice, you are stuck with the first
night’s room charge. The hotel, however, is likely stuck with the remaining six days of an empty room from
your one-week reservation.
Indeed, the hotel has an ongoing
parade of eventually resolved room
sales. By the time it knows what is
happening on Tuesday, the confusion
of Wednesday’s occupancy is about to
be resolved. Again, payments to the
hotel are consistently eventual.