Friday, February 20, 2015

Lessons in how not to develop software

Once in a while a great case study comes along that is a great teaching vehicle.   This is one of them.

Two current magazine articles cover the dismal debut last fall (2013) of www.healthcare.gov , America’s federal website where citizens can sign up for new and improved healthcare insurance under the Affordable Care Act (aka “Obamacare”).  “Code Red” is the title of the article in the March 10th, 2014 issue of TIME magazine.  The other, “Failure to Launch”,  is in the March/April, 2014 issue of Foreign Affairs.  I found a lot of useful information in them, and believe you’ll find their story extremely useful as an object lesson in quality control, governance, design, and rock-solid delivery.

Early in 2013, the Congressional Budget Office projected the enrollment of 7 million people between www.healthcarel.gov’s going live and the end of the year.  The website went live on October 1st.  Over the next 6 weeks, only a few thousand people were able to register.  Thousands could not navigate beyond the log-in dialog.  Thousands more were blocked by unhelpful error messages.  The entire system crashed several times in October and November.   A team of mostly Silicon Valley engineers (from Google, Oracle, and Red Hat), hastily and urgently selected on the strength of peer recommendation, spent October and November fixing hundreds of bugs and re-factoring lots of code.   I heard this afternoon on National Public Radio that over 6 million people have enrolled to date – certainly better than a few thousand, but roughly 3 months behind schedule.    What did the team of engineers and testers learn had gone wrong in this massive project that, in October, came within a day or two of being junked?

Here’s a short list of contributing problems.   Some look very familiar.
·         Insufficient testing.  In late March, 2013, McKinsey prepared a report for the Department of Health and Human Services (parent organization for the Center for Medicaid and Medicare Services (CMS), who had been assigned ownership for producing the website – the actual development was done by CGI Group and other contractors) that pointed to “insufficient time and scope of end-to-end testing”. 
·         Insufficient performance testing.    Also in March-April, CMS warned its management that they would not be able to test load adequately by October because they could not simulate so many people logging on simultaneously. 
·         Failure to report correct tested capacity.   In October, the team of Silicon Valley coders and testers reported that the website was crashing because its capacity was not 200,000 concurrent users, as had been reported, nor 100,000, but at most a mere 2,000 users.   The system crashed in October when 250,000 people tried to enroll.     
·         Reluctance to communicate failure up the management ladder.  The head of the Department of Health and Human Services did not know how bad the system was until after it had launched.   Neither did President Obama, since he looked to her for current status. 
·         Senior managers “drank the Kool-Aid”, repeating to themselves and to the press that everything was fine, without consulting the engineers who were aware of many problems (for example, page-load time was 8 seconds).   In September, David Simas, the White House senior adviser for communications, assured a reporter for Time magazine that “everything has been tested and is working perfectly.”  
·         No technical leadership.   Contractors bickered amongst themselves.  
·         No dashboard.   It was impossible to get a global view of progress or quality.
·         No sense of urgency among contractor employees even after the website had become the butt of jokes in world-wide media. 
·         Scope for go-live was global, not incremental.  It failed early, and colossally.
·         Caching was not used, so all queries were going directly to the database.
·         Tight coupling.   If something slowed down in one part of the backend, everything slowed down.


Apart from root-cause analysis and fixing over 500 bugs, what single practice was credited with saving the website?  Stand-ups.  Stand-ups, that is, as organized by Mikey Dickerson, a Google engineer with the title “Site Reliability Engineer”.   They aren’t the stand-ups we’re used to:
·         Twice-daily standups were held,  at 10 a.m. and 6:30 pm, 7 days a week, with average duration around 45 minutes.
·         To connect people at remote locations, open lines were left open 24 hours a day so that people could immediately exchange information.
·         Rules were posted outside the control center:

Rule # 1:  “The war room and the meetings are for solving problems.  There are plenty of other venues where people devote their creative energies to shifting blame.”

Rule #2:   “The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank.  If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it.” 


Rule #3:   “We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours.”  

No comments:

Post a Comment