These events can be queried, joined,
and connected based on their attributes. You can create new events by extracting attributes from a single event
or from a join across multiple events.
An event is an immutable set of at-
tributes with an identity.
The Quest for Identity
Some of today’s most challenging
problems come from the quest for
identity. Product matching, data science, fraud detection, homeland security, and more all struggle with figuring
out when one thing is the same as another thing so identity can be assigned.
Product matching is finding identity.
As discussed, providing an integrated
marketplace for stuff sold by wildly
disparate merchants is a big challenge.
The core of this challenge is matching
different SKUs from different merchants with different descriptions to
find the same product identity.
This is often made easier with UPC
or ISBN codes that actually do match.
This leaves the product-matching system with the easier job of comparing
attributes to verify identity. Product
matching is not always given the boost
from shared unique identifiers, and the
problem becomes a task of data science.
Data science is finding identity. In
data science, there are many objects,
each with many attributes. Each object
has a unique identity.
˲ Attaching new attributes: By comparing many objects and their attributes, the data-science algorithm associates new attributes with existing
˲ Merging object identities: By examining the attributes bound to sets
of objects, the data-science algorithm
can realize two objects are one. That, in
turn, unites their attributes.
Repeating the attribute/merge pattern
causes a new understanding of identity.
Fraud detection is finding identity.
Banks issuing credit cards invest heavily in fraud detection, as do retailers
and other institutions that accept
credit cards. Very large companies that
accept credit cards have a strong incentive to detect fraud since their banks
will charge them lower fees if their rate
of fraud is noticeably lower. Fraud detection is big business.
Fraud detection works by looking
at the transactions as objects with as-
Using Identity to Learn
Data science is based on identities, objects, and attributes. It has been used
to learn surprising new things. Identities are key to its work.
Data science and observations. Data
science revolves around identities. The
identities have attributes. It is the manipulation of these identities and attributes and comparison with other identities that share those attributes that leads
to new and deeper understanding.
Identities, objects, and attributes.
When observations are made, they are
stored as objects and given identities.
These objects have attributes. Analyzing the objects may lead to additional
attributes being added to them. Continued pattern matching on attributes
over large collections of objects can
lead to new attributes slapped onto the
sides of the objects.
Sometimes, looking at patterns on
the objects and their attributes leads
to new objects showing the connections between existing objects. This
will result in new identities for the new
objects. So, the pattern of attributes
becomes an identity in its own right,
which may lead to new attributes.
Attributes on identities—rinse and
repeat. It is the continuous cycle of
looking at lots and lots of attributes
on the objects and their identities
that leads to more attributes. These
new attributes are either attached to
existing objects or used to generate
new objects with their own independent identities.
Data science uses identities to achieve
Big Data Is Lots of Identities
Big-data systems such as MapReduce,
2 Apache Hadoop (http://hadoop.
apache.org), and Apache Spark (https://
spark.apache.org) take immutable inputs and apply functional transformations to produce immutable outputs.
Because of the immutable nature of
the inputs and outputs, it is easy to reason about fault tolerance when pieces
of the work fail and are restarted.
Each of these big-data systems leverages the identities of data items to connect work and storage spread across
MapReduce and Apache Hadoop.
These big-data systems look at the datasets they process as a bunch of key/
value pairs. Consider MapReduce and
˲ The map function of MapReduce
takes a series of key/value pairs and
makes a set of output key/value pairs.
These output pairs may be the same as
or different than the map function input.
˲ The reduce function is called once
for each unique key and can iterate
through the values associated with that
key. There may be multiple values for a
Queries, joins, and more with keys.
Queries and joins in these big-data
environments leverage the keys in the
key/value pairs. These are sorted across
shards with the map function. The queries and joins are applied by the reduce
function handling all key/value pairs
with the same key (or identity).
Because the map function can arrange an input key/value into another
shaped key/value, MapReduce and Hadoop can query, sort, and join on arbitrary fields in the data. Putting the join
fields into the key and sorting allows
for a huge flexibility in function.
Big-data means handling lots of keys.
Big-data systems require handling lots
of keys. They can be spread around in a
scalable fashion across very large clusters of servers to accomplish massive
scale. The identity provided by the keys
hooks it all together.
The “Internet of Identities.” IoT, or
the Internet of Things, is the new trend
wherein massive numbers of events
from disparate devices are processed
at high rates.
Internet of Things: Identifying the
thing. In Io T, an extremely large number of devices that may barely qualify as
computers generate massive numbers
of events to be processed. Each of these
devices will have an identifier in some
form. As it generates events, each of
these events will have a more detailed
identifier that usually specifies its device of origin.
Each of these events will, in turn,
have a bunch of attributes that are specific to the device. Events originating
from your refrigerator will have different attributes than events originating
from your car’s transmission or from a
security camera at a large stadium.
Querying, joining, and connecting
things. Similar to what is seen in big
data, each of these Io T events has an
identity and a bunch of attributes.