The application must include mechanisms to ensure that these, too, are
idempotent. This means remembering
in some fashion that the message has
been processed so that subsequent attempts make no substantive change.
Remembering messages as state. To
ensure the idempotent processing of
messages that are not naturally idempotent, the entity must remember they
have been processed. This knowledge
is state. The state accumulates as messages are processed.
In addition, if a reply is required, the
same reply must be returned. After all,
you don’t know if the original sender
has received the reply.
Manage state for each partner. To
track relationships and the messages
received, each entity within the scale-agnostic application must somehow
remember state information about its
partners. It must capture this state on
a partner-by-partner basis. Let’s name
this state an activity (see Figure 6).
Each entity may have many activities
if it interacts with many other entities.
Activities track the interactions with
Each entity consists of a set of activities and, perhaps, some other data that
spans the activities.
Consider the processing of an order consisting of many items for purchase. Reserving inventory for shipment of each
separate item will be a separate activity.
There will be an entity for the order and
separate entities for each item managed
by the warehouse. Transactions cannot
be assumed across these entities.
Within the order, each inventory
item will be separately managed. The
messaging protocol must be separately
managed. The per-inventory-item data
contained within the order entity is an
activity. While it is not named as such,
this pattern frequently exists in large-scale apps.
In an almost-infinitely scaled appli-
cation, you need to be very clear about
relationships. You can’t just do a query
to figure out what is related. Everything
must be formally knit together using
a web of two-party relationships. The
knitting is done with the entity keys.
Because the partner is some distance
away, you have to formally manage your
understanding of the partner’s state
and reordering. It introduces the notion
of an activity as the local information
needed to manage a relationship with
a partner entity.
Retries and idempotence. Since any
message may be delivered multiple
times, the application needs a discipline to cope with repeated messages.
While it is possible to build low-level
support for the elimination of duplicate messages, in an almost-infinite
scaling environment the low-level support would need to know about entities. The knowledge of which messages
have been delivered to the entity must
travel with the entity when it moves because of repartitioning. In practice, the
low-level management of this knowledge rarely occurs; messages may be
delivered more than once.
Typically, the scale-agnostic (
higher-level) portion of the application must
implement mechanisms to ensure that
the incoming message is idempotent.
This is not essential to the nature of the
problem. Duplicate elimination could
certainly be built into the scale-aware
parts of the application. This is not yet
available. Hence, let’s consider what
the poor developer of the scale-agnos-
tic application must implement.
Defining idempotence of substantive
behavior. The processing of a message is
idempotent if a subsequent execution of
the processing does not perform a
substantive change to the entity. This is an
amorphous definition that leaves open
to the application the specification of
what is and what is not substantive.
If a message does not change the
invoked entity but only reads information, its processing is idempotent.
This is true even if a log record describing the read is written. The log record
is not substantive to the behavior of
the entity. The definition of what is
and what is not substantive is application specific.
Natural idempotence. To accomplish
idempotence, it is essential that the message does not cause substantive side
effects. Some messages provoke no
substantive work any time they are processed. These are naturally idempotent.
A message that only reads some
data from an entity is naturally idempotent. What if the processing of a
message does change the entity but not
in a substantive way? Those, too, would
be naturally idempotent.
Now it gets more difficult. The work
implied by some messages actually
cause substantive changes. These messages are not naturally idempotent.
Figure 6. Activities are what an entity
Figure 7. Example of one activity per partner.