Launch It Like It’s Hot
If an AFP will have significant resource
needs, load testing is important. Everyone knows how to do load testing:
simulate thousands of HTTP requests
and take measurements. Find and fix
the bottlenecks and repeat until you
You also need to plan for the situation where the AFP goes viral and
receives 10 times or 100 times more
users than you could ever expect. The
easy strategy here is simply to plan on
disabling the AFP, but it would be disappointing that the reward for success
was to turn the feature off.
Fixing such a situation is difficult
because normal solutions might take
weeks to implement and April Fools’
Day lasts only one day. If you fix a problem and relaunch the next day, you
have missed the boat.
Facebook is in a similar situation
when launching real features because
there is a lot of press around a new
feature and Facebook needs to “get
it right” on the first try. When Facebook was new, growth was slow and
bottlenecks could be fixed by simply fixing them at the pace Facebook
was growing. By 2008 Facebook had
millions of users, and a new feature
would go from 0 to millions of users
within hours. There would be no time
to fix unexpected bottlenecks. A failed
launch is highly visible and embarrassing, often becoming front-page
news. There is no way, however, to
build an isolated system big enough
to perform load testing.
To solve this problem, Facebook
uses a technique called a “dark
launch:” testing a feature by first
launching it invisibly. For example,
Facebook launched Chat six months
early but made it invisible (CSS display:
code was in your browser, but it did
not display itself. A certain percent-
age of users received a signal to send
simulated chat messages through the
system. The percentage was turned
up over time so that developers could
spot and fix any performance issues. By
the time the feature was made visible
(and the test messages were disabled),
Facebook’s engineers were confident
that the launch would not have perfor-
mance problems. It is suggested that
nearly every feature that Facebook will
launch in the next six months is already
running in your browser. 4
Google did something similar be-
fore launching IPv6 connectivity; your
browser was running invisible Java-
Script that tested whether your ISP con-
nection would fail if IPv6 were enabled.
Worries were for naught, but the test
increased confidence before launch.
Stack Overflow dark launches
new ad-serving infrastructure. When
launching major features, we first use
the system to transmit house ads that
are invisible to users. Once performance is verified, we make the advertisements visible. Sadly, we did not use
this technique when launching StackEgg, but now we know better.
Pranks with Minimal
Technical issues can be avoided with
proper testing, but there is a strategy that avoids the issue altogether.
Simply create a prank that has no
operational impact, or directs the
The “Dance Dance Authentication”
example is one such prank. The prank
was simply a blog post and a link to a
YouTube video (https://www.youtube.
com/watch?v=VgC4b9K-g YU). This does
not entirely avoid the issue, but if your
success ends up overloading You Tube’s
network, at least it is not your problem.
You can also simply take an existing feature and create an alternative
explanation or history for it. For example, you may have heard of “the
teddy bear effect.” Many have observed that often the act of asking
a question forces you to think out
enough details to realize the answer
yourself. In Bell Labs folklore there
was a researcher known for helping
people with research roadblocks.
People would go to him for suggestions. By listening, they would come
up with the answer themselves. Once,
he left on a long vacation and left a
teddy bear on his desk with a note
that read, “Explain your problem to
the bear.” Many people found it was
equally effective. (Lately, the Internet
has started calling this “the rubber
Suppose you run a question-and-answer website: some users post questions, and other people post answers.
Suppose also that the website has a fea-
ture that permits people to write up the
answers to their own questions. A very
simple but effective AFP would be to
rename this feature “teddy bear mode”
and write a blog post claiming this to
be an entirely new feature, based on
the power of a teddy bear’s ability to
help solve technical issues.
Successful AFPs require care and planning. Write a design proposal and a
project plan. Involve operations early.
If this is a technical change to your
website, perform load testing, preferably including a “dark launch” or hidden launch test. Hide the prank behind
a feature flag rather than requiring a
new software release. Perform a retrospective and publish the results widely.
Remember that some of the best
AFPs require little or no technical
changes at all. For example, one could
simply summarize the best practices for launching any new feature
but write it under the guise of how
to launch an April Fools’ prank. That
would be hilarious.
Ray Tracing Jell-O Brand Gelatin
Paul S. Heckbert
The Burning Bag of Dung—and Other
10 Optimizations on Linear Search
Thomas A. Limoncelli
1. Dumke-von der Ehe, B. The making of StackEgg;
2. Kottasova, I. Google’s April Fools’ prank backfires big
time. CNNtech; http://money.cnn.com/2016/04/01/
3. Pike, K. Stack Overflow unveils the next steps in
computer security. Stack Overflow Blog; https://
4. Rossi, C. Pushing millions of lines of code five days a
week. Facebook, 2011; https://www.facebook.com/
5. Stack Exchange Network Status. Outage
postmortem: March 31, 2015; http://stackstatus.
Thomas A. Limoncelli is the co-editor of the book,
“The Complete April Fools’ Day RFCs”
( http://www.rfchumor.com/), and is the site reliability
engineering manager at Stack Overflow Inc. in
New York City. He blogs at EverythingSysadmin.com
and tweets Yes That Tom.
Copyright held by author.
Publication rights licensed to ACM. $15.00.