Once in a while a great case study comes along that is a
great teaching vehicle. This is one of them.
Two current magazine articles cover the dismal debut last
fall (2013) of www.healthcare.gov ,
America’s federal website where citizens can sign up for new and improved
healthcare insurance under the Affordable Care Act (aka “Obamacare”). “Code
Red” is the title of the article in the March 10th, 2014 issue
of TIME magazine. The other, “Failure to Launch”, is
in the March/April, 2014 issue of Foreign Affairs. I found a lot
of useful information in them, and believe you’ll find their story extremely
useful as an object lesson in quality control, governance, design, and
rock-solid delivery.
Early in 2013, the Congressional Budget Office projected the
enrollment of 7 million people between www.healthcarel.gov’s
going live and the end of the year. The website went live on October 1st.
Over the next 6 weeks, only a few thousand people were able to register.
Thousands could not navigate beyond the log-in dialog. Thousands more
were blocked by unhelpful error messages. The entire system crashed
several times in October and November. A team of mostly Silicon
Valley engineers (from Google, Oracle, and Red Hat), hastily and urgently
selected on the strength of peer recommendation, spent October and November
fixing hundreds of bugs and re-factoring lots of code. I heard this
afternoon on National Public Radio that over 6 million people have enrolled to
date – certainly better than a few thousand, but roughly 3 months behind
schedule. What did the team of engineers and testers learn
had gone wrong in this massive project that, in October, came within a day or
two of being junked?
Here’s a short list of contributing problems.
Some look very familiar.
·
Insufficient testing. In late
March, 2013, McKinsey prepared a report for the Department of Health and Human
Services (parent organization for the Center for Medicaid and Medicare Services
(CMS), who had been assigned ownership for producing the website – the actual
development was done by CGI Group and other contractors) that pointed to
“insufficient time and scope of end-to-end testing”.
·
Insufficient performance testing.
Also in March-April, CMS warned its management that they would not be
able to test load adequately by October because they could not simulate so many
people logging on simultaneously.
·
Failure to report correct tested capacity.
In October, the team of Silicon Valley coders and testers reported that the
website was crashing because its capacity was not 200,000 concurrent users, as
had been reported, nor 100,000, but at most a mere 2,000 users. The
system crashed in October when 250,000 people tried to enroll.
·
Reluctance to communicate failure up the
management ladder. The head of the Department of Health and Human
Services did not know how bad the system was until after it had
launched. Neither did President Obama, since he looked to her for
current status.
·
Senior managers “drank the Kool-Aid”,
repeating to themselves and to the press that everything was fine, without
consulting the engineers who were aware of many problems (for example,
page-load time was 8 seconds). In September, David Simas, the White
House senior adviser for communications, assured a reporter for Time magazine
that “everything has been tested and is working perfectly.”
·
No technical leadership.
Contractors bickered amongst themselves.
·
No dashboard. It was
impossible to get a global view of progress or quality.
·
No sense of urgency among contractor
employees even after the website had become the butt of jokes in world-wide
media.
·
Scope for go-live was global, not
incremental. It failed early, and colossally.
·
Caching was not used, so all queries were
going directly to the database.
·
Tight coupling. If something
slowed down in one part of the backend, everything slowed down.
Apart from root-cause analysis and fixing over 500 bugs, what
single practice was credited with saving the website? Stand-ups.
Stand-ups, that is, as organized by Mikey Dickerson, a Google engineer with the
title “Site Reliability Engineer”. They aren’t the stand-ups we’re
used to:
·
Twice-daily standups were held, at 10 a.m.
and 6:30 pm, 7 days a week, with average duration around 45 minutes.
·
To connect people at remote locations, open
lines were left open 24 hours a day so that people could immediately exchange
information.
·
Rules were posted outside the control center:
Rule
# 1: “The war room and the meetings are for solving problems. There
are plenty of other venues where people devote their creative energies to
shifting blame.”
Rule
#2: “The ones who should be doing the talking are the people who
know the most about an issue, not the ones with the highest rank. If
anyone finds themselves sitting passively while managers and executives talk
over them with less accurate information, we have gone off the rails, and I
would like to know about it.”
Rule
#3: “We need to stay focused on the most urgent issues, like things
that will hurt us in the next 24-48 hours.”
No comments:
Post a Comment