ing some of the same latency issues as
their newer HFT clients.
Collocation. For exchanges, collocation can be an invaluable source of revenue. The more incumbent exchanges
run their own data centers (such as
NYSE and NASDAQ), and customers
pay collocation fees to the exchanges
directly. Newer exchanges must collocate in third-party data centers that
house financial customers (for example, Equinix’s NY4 in Secaucus, NJ).
When exchanges operate inside
third-party hosting facilities, they are
subject to the same power and cooling
costs as their HFT brethren. As such,
many exchanges focus on delivering
high-throughput systems using standard x86 designs.
Networking. Exchange networking
is as challenging as HFT networking
but also has a deeper focus on security. Information arbitrage, or the
practice of gaining information about
a market that is not directly intended
for the recipient, is a major concern.
An example of information arbitrage
is when an exchange participant
“snoops” the wire to read packets
from other participants. This practice
is easily thwarted with deployment of
VLANs for each client.
Most exchanges still deploy 1GbE
at the edge. This is both a function of
history (change management in an exchange is a long process) and practicality. By putting 1GbE edge networks in
place, the exchange can protect itself
somewhat from the onslaught of messages by limiting both the inbound
bandwidth and adding a subtle trans-coding hit. For example, a 1GbE Arista
7048 has a three-µsec latency for a 64-
byte frame, which is a 350-nsec latency
for the same frame on a 7150 (10GbE).
Gateway. The gateway is the first exchange subsystem that client flow encounters. Gateways have evolved over
the years to take on more responsibility, but at the core they serve as feed
handlers and tickerplants. The gateway
receives a client request to trade (
historically in the FIX format, but as latency became paramount, exchanges have
switched to proprietary binary protocols). It then translates the request to
a more efficient internal protocol, and
routes the request to the appropriate
underlying exchange matching engine.
The gateway also serves as the first
line of defense for erroneous trades.
For example, if a client attempts to buy
AAPL for $5,000 (whereas AAPL is offered at $461), the gateway can reject
Market Data Gateway. Traditionally
the Order Gateway (which receives client requests to trade) and the Market
Data Gateway (which distributes market-data feeds) are two separate systems, and often two separate networks.
For market-data distribution, two
methods are common: User Datagram
Protocol (UDP) Multicast for collocated customers; and Transmission Control Protocol (TCP) for noncollocated
customers. Customization takes place
here as well (for example, NASDAQ’s
SoupTCP9). In some markets (for example, FX), all market data is distributed over TCP in Financial Information
Exchange (FIX). For the other markets,
data is often distributed over UDP in a
binary format or an easy-to-parse text
format. The predominant two binary
formats are ITCH6 and OUCH,
both sacrifice flexibility (fixed-length
offsets) for speed (very simple parsing).
Gateways are often shared across
customers, as a gateway for each and
every exchange participant would likely
require a massive data-center footprint.
As such, gateways must be closely monitored for malicious manipulation. An
example of gateway “gaming” is shown
in Figure 11. In Figure 11a, client A is
connected to two distinct gateways. In
11b, Client A induces extreme load on
Gateway 1, causing Client B traffic to
slow. In 11c, Gateway 1, not under load,
slows all attempts for Client B to cancel
resting markets. Client A has an advantage with the self-made fast path.
Matching engine. The matching engine is the core of the exchange, and
like its HFT cousins is fairly straightforward. A matching engine, shown in
Figure 12, is a simple queue management system, with a queue of bids and
a queue of offers. When a customer
attempts to execute against a queue,
the exchange searches the queue until
it reaches the requested size and removes those orders from the queue.
The difficulties arrive in determin-
ing who receives notifications first. The
aggressing party does not know (for
certain) it has traded 10,000 shares un-
til receiving a confirmation. The pas-
sive parties do not know they have been
executed until they receive a confirma-
tion. Finally, the market as a whole
does not know the trade has occurred
until the market data is published with
the new queue. Problems such as this
are becoming increasingly more diffi-
cult to solve as we move from millisec-
onds to microseconds to nanoseconds.
The world of high-frequency trading is
rich with problems for computer scientists, but it is fundamental to the new
marketplace of automated trading,
which is responsible for the majority of
transactions in the financial markets
today. As HFT moves from milliseconds to microseconds to nanoseconds,
the problems becoming increasingly
more difficult to solve, and technology
must strive to keep up.
nUMA (non-Uniform Memory Access):
FPGA Programming for the Masses
David Bacon, Rodric Rabbah and Sunil Shukla
Computing without Processors
1. arista networks. 7124FX application switch; http://
2. arista networks; 7150 series 1/10 gbe sFP ultra-low
latency switch; http://www.aristanetworks.com/en/
3. Cbs news. 2010. robot traders of the nyse. sixty
minutes overtime; http://www.cbsnews.com/video/wa
4. millar, m. ‘lightning fast’ future traders working in
nanoseconds. bbC news, 2011; http://www.bbc.co.uk/
5. moyer, l. and lambert, e. Wall street’s new masters.
Forbes (sept. 21, 2009), 40–46; http://www.forbes.
6. nasDaQ. nasDaQ totalView-ItCh 4. 1, 2013; http://
7. nasDaQ. omX Co-location; http://app.qnasdaqomx.
8. nasDaQ. ouCh Version 3. 1, 2012; http://www.
9. nasDaQ. souptCP; http://www.nasdaqtrader.
Jacob Loveless is the Ceo of lucera, a new york-based
cloud computing company that services low-latency
financial customers. he was a special contractor for the
u.s. Department of Defense with a focus on heuristic
analysis of things that cannot be discussed.
© 2013 aCm 0001-0782/13/10 $15.00