Share this story with your colleagues now.
In TIME Magazine's cover story this week, author Steven Brill offers a behind-the-scenes look at the nightmare launch of HealthCare.gov—and the high-tech rescue team that was able to save it.
How the White House—and Zients—say they fixed a badly broken site
Nearly three weeks after the website's troubled launch, it had become clear that the early explanation for the problems—sky-high user volumes—was "anything but the whole story," Brill writes. Jeff Zients, former deputy director of the Office of Management and Budget, and White House Technology Officer Todd Parks were tasked with "finding fresh eyes who could decide whether the thing was salvageable," Zients recalls. (Zients is a former CEO of The Advisory Board Company.)
The duo recruited a group of unknown—except in elite technology circles—coders and troubleshooters working in various enterprises across the country. Many of those unknowns, including Civis Analytics' Gabriel Burt and Google's Mikey Dickerson, were former "Obama-campaign whiz kids" who helped develop one of the most accurate voter targeting models ever used in a national campaign.
Over the next six weeks, the team worked furiously to meet a self-imposed Dec. 1 deadline to fix the flailing website. They were put on the payroll of contractor QSSI—one of the Washington contractors tasked with building the site—as hourly workers, making "a fraction" of their usual pay, recalls Dickerson, who was appointed team leader of the effort.
Dickerson, who held twice-daily "stand-up" meetings at 10 a.m. and 6:30 p.m., only imposed three rules throughout the process. Those rules were posted outside the group's war room:
- Rule 1: "The war room and the meetings are for solving problems. There are plenty of other venues where people devote their creative energies to shifting blame."
- Rule 2: "The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. If anyone finds themselves sitting passively while managers and executives talk over them with less accurate information, we have gone off the rails, and I would like to know about it."
- Rule 3: "We need to stay focused on the most urgent issues, like things that will hurt us in the next 24-48 hours."
According to Brill, the "identify problem, solve problem, try again" culture was an important part of the rescue squad's work ethic. Their first order of business was to create a dashboard to measure what was happening on the website, including how many people were using it, what the response times were for various click-throughs, and where traffic was getting tied up. The task took about five hours of coding on the first night.
However, the team encountered hiccups along the way—most notably, a 37-hour website outage on Oct. 27 and a 40-hour outage on Oct. 30, the same day HHS Secretary Kathleen Sebelius testified on Capitol Hill about the website's troubles before a House Subcommittee.
The latest on HealthCare.gov
"Those outages were totally demoralizing," Burt said, adding, "We thought we were on our way. We had gotten some momentum but lost it." Nonetheless, the group forged ahead, increasing its number of website upgrades from three or four in October to more than 25 in November.
According to Zients, "The team ran two-minute drills to perfection. We had the best players on the field. Some plays didn't work. We talked about some of those. But there was never any finger pointing. People just hustled right back to the line, and we ran the next play."
A Thanksgiving turnaround
The team increased its ranks in November as it sought to address the long list of errors that had plagued the website. Jini Kim, a former Google employee, was nicknamed the "Queen of Errors" for reducing the website's 6% error rate—just 15 or 16 clicks would turn up a problem—to 0.5%.
More good news came on Dec. 1, when Zients issued a public report card showing the website's turnaround: The system was now able to handle 50,000 users at a time; more than 400 bugs had been fixed; and uptimes had gone from 43% at the beginning of November to 95%.
More from Brill: Longest article in TIME history explains high U.S. health costs
In about a tenth of the time that it had taken Washington contractors to spend over $300 million building a site that didn't work, the team had fixed it, Brill writes. As of its mid-February report covering the period through Jan. 31, CMS says the site had processed 1.9 million enrollments.
Brill notes that every member of the team told him the same thing—that the project was the toughest and most rewarding experience of their lives. Burt told TIME, "The two months I spent on this were harder and more intense than the 17 months I spent on the campaign, but I loved every minute of it ... I believe in getting people health care. I am so proud of this" (Brill, TIME, 3/10).
Next in the Daily Briefing
Daily roundup: March 3, 2014