By Asami Novak

Two years ago, at exactly the same time New Relic was hosting its inaugural FutureStack conference, our founder and CEO Lew Cirne got an unusual email. The subject line said, “WH needs help,” and the message was followed by a call from the White House’s technical SWAT team, asking if New Relic could come in and help (what was then) a rather messy and complicated HealthCare.gov situation.

According to Lew, “It was the mother of all APM problems” that, fortunately, turned into a remarkable success story. And that was all thanks to the people on the front lines, those individuals who were part of President Obama’s fix-it team, including Sean O’Neil, senior director of Enterprise Architecture at Optum Technology, who—along with Apple co-founder Steve Wozniak—was one of the keynote speakers opening Day 2 of this year’s FutureStack.

O’Neil gave a heartfelt and entertaining behind-the-scenes account of what went down, and, more important, how the changes his team made in order to fix HealthCare.gov are today transforming the way government manages IT.

So what exactly went wrong?

“Generally speaking, when people ask that question, they want to hear about all the stupid things people did,” said O’Neil. “But the truth is, there was no one stupid thing that someone did … just a whole bunch of stuff you’ve probably run into before.” In spite of the lengthy list of technical problems with the site, the biggest culprit, according to O’Neil, was a lack of process. “To change the technology environment, we first needed to change the culture of the environment,” he said. And that led to a series of culture and process changes.

Culture > technology

According to O’Neil, changing the culture in the Healthcare.gov .war room was more important than the actual technology changes the HealthCare.gov team later implemented. The process and culture changes included:

• Not making perfection the goal
• Problem-management on a 12-hour cycle
• Viewing diversity in thought as a strength
• Avoiding passive voice (and passive thinking!) to communicate

These changes then led to three core rules of the HealthCare.gov .war room:

Rule #1: The .war room is for solving problems, not shifting blame.

Rule #2: The ones who should be doing the talking are the people who know the most about an issue, not the ones with the highest rank. (This point, in particular, got hearty applause from FutureStack15 attendees.)

Rule #3: We need to stay focused on the most urgent issues—things that’ll hurt the team in the next 24-48 hours.

“The problems that plagued the HealthCare.gov website weren’t anything special, but how we fixed it was,” said O’Neil. “The culture changes we put in place are now rippling through government in many exciting ways.”

U.S. government joins the 21st century

Thanks to the lessons learned from HealthCare.gov, the rest of the U.S. government now has a set of best practices to follow as it continues down its journey toward digital transformation, O’Neil said. Among the new technologies being used, he said, are public and hybrid clouds, Amazon EC2, MySQL, New Relic APM and New Relic Insights, Node.js and Nginx, Jenkins, and Puppet, among others.

New Relic Plugins were particularly helpful in driving technology advances at HealthCare.gov. As O’Neil described it, he told the team that, “Instead of being slaves to crappy, expensive, off-the-shelf software, you can now write your own plugins, use New Relic, and save taxpayers a lot of money.”

As a final note, O’Neil encouraged members of the audience to get involved and join forces with the U.S. Digital Service, a new government agency that was formed by some of the original members of the HealthCare.gov team, including Mikey Dickerson. With a focus on redefining how the federal government works for American citizens, O’Neil told the crowd, the U.S. Digital Service presents an incredible opportunity to “put your skills to work and help change the way our government does IT.”

About the Author

Asami Novak is senior manager of content strategy and development at New Relic. Prior to joining the New Relic team, she wrote marketing and ad copy for a variety of B2B and B2C companies. Her editorial writing has appeared in WIRED and Dwell, among other publications. View posts by Asami Novak.

Sean O’Neil photo © Andrew Weeks Photography.