ProPublica

Journalism in the Public Interest

Cancel

Here’s Why Healthcare.gov Broke Down

Federal officials have pointed to overwhelming demand to explain the site’s problems. But web developers, other experts and journalists have uncovered more fundamental issues with the design and functioning of the site.

(Karen Bleier/AFP/Getty Images)

For the past two weeks, healthcare.gov, the federal government’s new health insurance marketplace, has been bogged down by problems, preventing users (including me) from viewing insurance options and plans on the website.

Federal officials have pointed to overwhelming demand to explain the site’s problems. But web developers, other experts and journalists have uncovered more fundamental issues with the design and functioning of the site.

Here are excerpts from five of the better stories explaining what happened:

Healthcare.Gov’s Flaws Found, Fixes Eyed

By Christopher Weaver and Louise Radnofsky, The Wall Street Journal

Much of the problem stems from a design element that requires users of the federal site, which serves 36 states, to create accounts before shopping for insurance, according to policy and technology experts. The site, healthcare.gov, was initially going to include an option to browse before registering, but that tool was delayed, people familiar with the situation said.

The decision to move ahead without that feature proved crucial because, before users can begin shopping for coverage, they must cross a busy digital junction in which data are swapped among separate computer systems built or run by contractors including CGI Group Inc., the healthcare.gov developer; Quality Software Services Inc., a UnitedHealth Group Inc. unit; and credit-checker Experian PLC.

If any part of the web of systems fails to work properly, it could lead to a traffic jam blocking most users from the marketplace. That’s just what happened: On Oct. 2, officials identified a bottleneck where those systems intersect at a software component sold by Oracle Corp. that still hasn’t been cleared.

Tech experts wary of more Obamacare glitches

By Brett Norman and Jason Millman, Politico

Some software engineers have suggested that the consumer end of the website, designed by one contractor, is not “talking to” the back end of the website, developed by a different company.

Diagnostic tools in Web browsers have identified coding issues that may be complicating account creation. The Wall Street Journal reported Friday that the administration is considering an overhaul of the registration system this weekend to allow people to browse health plan options without first creating an account. The paper said the tech experts are focused on a bottleneck where a flood of data meets an Oracle software component involved in identification verification.

From the Start, Signs of Trouble at Health Portal

By Robert Pear, Sharon LaFraniere and Ian Austen, The New York Times

Confidential progress reports from the Health and Human Services Department show that senior officials repeatedly expressed doubts that the computer systems for the federal exchange would be ready on time, blaming delayed regulations, a lack of resources and other factors.

Deadline after deadline was missed. The biggest contractor, CGI Federal, was awarded its $94 million contract in December 2011. But the government was so slow in issuing specifications that the firm did not start writing software code until this spring, according to people familiar with the process. As late as the last week of September, officials were still changing features of the Web site, HealthCare.gov, and debating whether consumers should be required to register and create password-protected accounts before they could shop for health plans.

Some say health-care site’s problems highlight flawed federal IT policies

By Craig Timberg and Lena H. Sun, Washington Post

The U.S. government spends more than $80 billion a year for information-technology services, yet the resulting systems typically take years to build and often are cumbersome when they launch. While the error messages, long waits and other problems with www.healthcare.gov have been spotlighted by the high-profile nature of its launch and unexpectedly heavy demands on the system, such glitches are common, say those who argue for a nimbler procurement system.

They say most government agencies have a shortage of technical staff and long have outsourced most jobs to big contractors that, while skilled in navigating a byzantine procurement system, are not on the cutting edge of developing user-friendly Web sites.

These companies also sometimes fail to communicate effectively with each other as a major project moves ahead. Dozens of private firms had a role in developing the online insurance exchanges at the core of the health-care program and its Web site, working on contracts that collectively were worth hundreds of millions of dollars, according to a Government Accountability Office report in June.

How The First Internet President Produced The Government’s Biggest, Highest-Stakes Internet Failure

By Alex Howard, Buzzfeed

The debacle is merely the most visible example of how $80 billion spent annually by the federal government on information technology falls far short of delivering the quality or service any private company would expect at a fraction of that cost.

...

At the heart of the federal IT crisis is a complex system of regulations that rewards contractors that are better at bidding on giant federal contracts than at building software. While the political figures who commission or oversee those contractors are ultimately culpable, the work itself is done by the private sector. That’s not only true of civilian agencies, as the world was reminded when a private contractor for the National Security Agency, Edward Snowden, leaked key documents from the government and gave then to the press.

Finally, explore the contractors who worked on healthcare.gov and their campaign contributions, courtesy of the Sunlight Foundation.

I wonder why the designers didn’t envision a system that had a central mainframe with ‘X’ amount of satellite ‘stand-a-lone’ mini-frames. IMO, had this design been used the following would be possible:

1. Placement of the stand-a-lones would allow for a reduction of the demand to the mainframe. More stand-a-lones in dense populations would ensure that no crashes would occur, and if they did, it would be to those units alone, not the entire system.

2. Each stand-a-lone, with the capability to fully process every applicant, would use a backdoor to file those applications to the mainframe. As for backup, each stand-a-lone would hold on station a copy of the application.

3. It is still possible to do this design and save the integrity of the Health Department’s effort to enroll a maximum of people.

Just saying…

It would have also been nice to see if there were contributions to Republican campaigns.

Other than scraping (ironically) other news services, Pro Publica - champion of leaked documents, publisher of “misplaced” IRS documents and champion of FOIA has nothing to say about a government Web site that won’t provide prices on mandatory health insurance unless users disclose names, under threat of prosecution for lying to federal agents and felony illegal access to computer systems?

Isn’t it an editors job to ask “Why?” Why did the administration create a Web site where the media can’t scrape insurance rates and systematically compare them to pre-ObamaCareless insurance rates? Why did US Dept. of Health and Human Services revise 45 CFR Parts 144, 147, 153, 155, and 156 to all sharing of information collected by the aforementioned Web site where all uses must log in?

I’m wondering how much distraction and tampering with decision processes were orchestrated by meddlings coming from Republicans and other “concerned” groups as the A.C.A. moved toward implementation?  Seems to me that an administration distracted with an avalanche of interference might easily have made some errors in judgement relative to the complexity of the task at hand.  Then again, there is a certain level of technological ineptitude imbedded in the machinery of our national and state governments.

The main problem is fairly simple to explain.  I gave a presentation last February citing a nationally known consultant who noted that most big IT projects - whether public or private - involve one new system that is then aligned with a number of existing systems.  In this case, however, several new systems were being developed simultaneously.

Additionally, the interfaces with existing systems are more numerous and difficult than people realize.  Just one example, California has three Medicaid systems running - not one - and each of the three is relatively old using old languages and running in many cases on old hardware.  The federal data hub is to connect five major federal systems with an untold number of state systems. These systems also need to connect with multiple insurance company systems - nearly all of which needed to be extensively modified.

The reason several major insurers opted out of the public exchanges is that the upfront costs to develop systems that would work with new and unproven federal and state systems is significant with little assurance that the additional policies sold would cover the upfront cost.
I am not motivated to attack the federal or state folks who have been trying to build this - other than to say I don’t think any of them really grasped the monumental task this would be nor really understood the high potential for multiple failures - even with much higher cooperation from many states.

Unfortunately they are now paying a huge price - politically and economically.

It is mind-boggling that the most powerful nation in the world doesn’t remotely have its IT act together. It sickens me to learn how much money we waste because we haven’t taken the time to streamline these systems yet the government has no problem tracking our every move or acquiescing to political lobbying.

@CR Most of the federal agencies have their IT set up. BATF, CIA, DEA DoD, FBI, NSA and more all have top of the line systems…

While the systems in Defense, and national security, and law enforcement have indeed been modernized and well funded - other systems are very antiquated - Social Security, IRS, Medicare, VA - and many of the state systems are quite old in both hardware and software.

For years he blamed George Bush..
Then he blamed an Anti-Islamic B Movie..
Then he blamed all of those ‘evil’ people that want less Gov’t intrusion..
Now he’s blaming ‘Overwhelming Traffic’..

He’s lucky he’s African American, or he’d be getting all the blame.

A few yrs ago under a previous administration VA rolled out a new computer system that was first used at a VA hospital in Tampa and it too crashed. Probably no oversight with contractors or the IT folks at the Fed are not the best!

Richard McGinnis

Oct. 17, 2013, 1:06 p.m.

” On Oct. 2, officials identified a bottleneck where those systems intersect at a software component sold by Oracle Corp. that still hasn’t been cleared. “
Java problem.

And while the usability of the exchange is a mess, it is masking any issues with the content (i.e., the plans being offered). On the NY exchange there are plans available at different price points that identical in terms of all the criteria listed on the site. There are other plans offered by companies with no history. Others where lists of physicians are not available. The notion of the marketplace requires not only a solid web platform but transparency and complete information when it comes to the plan choices available. Without good information, no market can succeed.

I wasn’t involved in any of this work, but I’ve worked on similar projects.  I can tell you exactly the reason the site broke:  Nobody asked for it to hold up to the load.

Seriously.  Some engineer, guaranteed, took the number of uninsured people and divided it by ninety days.  They looked at the server, and saw it could handle that number, and declared victory.

Nobody would think to write, in the specification, that the site would need to handle a million people in an afternoon, because we all see sites work well every day, so we all assume it’s free unless we’ve been bitten by it.

Honestly, that’s it.  It’s a hard problem to solve cheaply, and nobody’s going to solve that problem unless there are strict instructions to do so.  And someone needs to pay for it, too.  Look how much Google and Facebook pay to keep their architectures humming.  (Facebook, to its credit, has started posting information about its data centers under public licenses.)

If it had worked, we might be reading the article about how they spent twice as much on bandwidth as they needed to on the first day, and ten times the bandwidth since, what a waste of taxpayer dollars…

Again, I’m not a fan of Obama’s policy or ObamaCare, so I’m not defending the crown, here.  I just see these sorts of failures regularly, and it’s always the same story.  All of the cited problems boil down to “nobody asked.”

To put this into comparison, the “overwhelming demand” link cites 50,000 simultaneous users, whereas YouTube hits around 11,000 views per second.  I think that’s comparable, since they’re both sites that require sustained use, rather than a search engine.

But you see my point.  YouTube.  Why is healthcare.gov breaking under the load similar to what YouTube sees…?

@nickconfessore - Your spreadsheet would have been most meaningful if you would have a column for ‘Romney Campaign Contributions in 2012’.  Seems Your a Republican of the Ted Cruz type - just chaos and obstruction for no good reason.

And it is worth noting that its your kind that get’s the headline.

clarence swinney

Oct. 18, 2013, 8:31 a.m.

A recent local article blamed President Obama for our horrid debt
Debt incurred by President Obama
% of GDP—Fiscal year (ends 9-30 of each year.)
2009—-10.0%—GW Bush last budget
2010—-8.4%
2011—-8.3%
2012—-7.0%
2013.—4.0% (projected)
Source: cbo.gov
The major causes for our current 16.7B debt were two wars unfunded, two huge tax cuts,  Part D Medicare unfunded, Obama 800B stimulus and payroll tax cut.
Bush left a 11.9B Debt .a 6.1B increase on Clinton 5.8B.
.Obama added 4.8 B increase over Bush 11.9B. A dollar record for four years.
In fairness to Bush cbo predicted a 10,000B surplus in ten years so he felt the people could use the money. He did not predict 9-11-01. It was a mistake to get two wars. Part D was great for retirees.

I can’t believe 2 people on this forum have actually blamed Republicans for these glitches.  I had been gone from ProPublica because of thoughtless domogogery such as this.  The success or failure of the ACA is fully creditable to the Democrats.

Tom, I was thinking THAT same thing.  lol   There’s NOTHING about this debacle that’s the Republicans’ fault.  As a matter of fact, the Republicans did their best to prevent this absolute mess.

When it all falls apart, as it is on an hourly basis, the electorate will be thanking the Republicans for at least TRYING to stop this train wreck.

Here we go again, thinking and saying the GOP isn’t at fault when in reality no one knows who is at fault. The success or failure of the ACA isn’t alone the Democrats…it belongs to all in this country. Our private ran healthcare system is a failure and yet there are so many out there who insist it works as is. They are either blind or stupid to think it works. Instead of blaming someone, other than yourself, why don’t you start gathering and force our so called Congress to make changes. Obama has tried with absolutely no help from the GOP over the past several years. Wonder why the GOP doesn’t wish to help? I think it is because it wasn’t their idea and now they are acting like spoiled children. Remember as citizens of these country we can make changes, instead of only sitting around and blaming others.

steve hammill

Oct. 22, 2013, 9:56 a.m.

This should TERRIFY any thinking person.

$400 million for a website?
...okay, it’s gotta scale to millions of nearly simultaneous hits…but…......

Building a website that scales to millions of nearly simultaneous hits and interfaces with the disparate systems used in the 50 states IS TRIVIAL when compared to recreating a healthcare system in Obama’s image.

My senator Max understated the problem when he described it as “a train wreck.”

InterestedObserver2

Oct. 22, 2013, 10:31 a.m.

They “blamed a lack of resources.”  Because $50 million and three years just ain’t enough money or time to write an e-commerce software package, right?

Kat:

When Obamacare was SHOVED down America’s throat, it did not get one Republican vote.  It did not need one Republican vote.  The Democrats were a majority in the House and in the Senate.  Democrats were the ONLY ones who voted this in. 

In this particular case, yes, when this train wreck happens, I will ONLY blame the Democrats.  Usually I put the blame on both of them because I can’t stand all the ugliness in Washington. 

I think that whole Washington system needs to be revamped and take the wealth out if it, but that’s an entirely different discussion.

This whole Obamacare debacle is COMPLETELY the blame of the Democrats.

Charles Gerhards

Oct. 23, 2013, 4:09 p.m.

I am very suspicious that most of the glitches were caused by the Koch brothers or their cahoots. They could have bribed someone to sabotage the software. I am very tired of Mitch McConnell getting all the airtime to make his false statements about how the American public doesn’t like ObamaCare. A fairly crafted survey would prove him wrong.

Uurrgg, Oracle strikes again. Do some research and the cost of this failure is nothing like the billion dollar Air Force failure, also Oracle.

The cost of healthcare, people who sue, and lawyers are to blame.  Oh yeah, Obama is a lawyer, makes sense.

Andrew Menard

Nov. 2, 2013, 3:27 p.m.

I have believed from the get go that the roll out was sabotaged by the red states that did not open their own exchanges of their own shifting the burden to the Fed site and the GOP which refused to increase funding for the Federal site to accommodate the increased load.  The fact that the GOP shut down the government on the day that ACA launched is just frosting on the cake.

These beliefs have been hardened after reading an article on another blog, Politicusa “Republicans Deliberately Sabotaged the ACA Website, Hoping the Law Would Implode “.  The information there reenforced my belief that the fix was in.  It is just a sin that the main stream media has become so shallow as to not really do their jobs any more.

Andrew, even if what you say is true, which I don’t agree with, that doesn’t address the REAL issue.  The Republicans didn’t tell President Obama to lie and say, “If you like your insurance, you can keep your insurance PERIOD” and, “If you like your doctor, you can keep your doctor PERIOD.  I assure you.” 

And then now we find out that the President KNEW before even saying that that that was not going to be true for MANY Americans. 

So you can try to deflect blame all you want to Republicans, but there’s no way of defending President Obama’s false representations to shove this unworking debacle Obamacare down our throats.

Andrew Menard

Nov. 2, 2013, 3:55 p.m.

BevAg..I hold no politician to what is said in the heat of a campaign, I also would give the President a bit of leeway on this point as most of the cancellations are for policies that are not policies just a donation to an insurance company.  If the owner of said policies actually was in a major health crisis they would end up with the majority the bill and be force to go bankrupt to get back to normalcy.  No one ever said that you lose your doctor and I don’t really see how that could happen, unless you are in an HMO and that plan did not meet the standards of the law.  Standards under laws are everywhere in the nation, seat belts are a good example as is auto insurance.  The fact that a small percentage of the population is going to lose policies that are not at the new standard is the way thing sometimes go and don’t forget that the insurance companies are using this to increase their profits justly or unjustly, it is what they do.

I hold politicians to what they say ESPECIALLY when the President said people will be able to keep their present insurance plans when he KNEW that he was making regulations that would force those insurance plans to no longer be good.

That was an out-and-out lie with no defense.  That was not a Republican lying or misleading or whatever other deceptive name you want to put on it.  It was the President misleading, lying to get this VERY unfavorable legislation shoved down everyone’s throat. 

But I’m okay with it now.  At least this has leveled the playing field.  Even the incredibly left-leaning media can’t pretend he didn’t lie, and they’re not pretending he didn’t lie.  The only ones pretending that President Obama didn’t lie are people like yourself that would follow this President off a cliff.

Steve Hammill

Nov. 3, 2013, 1:43 a.m.

>>>I hold no politician to what is said in the heat of a campaign

Then what do you hold them accountable for?

Steve Hammill

Nov. 3, 2013, 1:53 a.m.

@Andrew

>>>I have believed from the get go that the roll out was sabotaged…

I’m happy the Pubs failed.
Now we get to see this plan work - or not.

This would be fun were it not for the real impact on my premiums, deductibles, and services available.

Politically speaking, I think ObamaCare violates the “if it ain’t broke, don’t fix it” axiom of the universe. ObamaCare is anything but Zen.

This article is part of an ongoing investigation:
Obamacare and You

Obamacare and You

The Rollout of the Affordable Care Act has been marred by glitches and political opposition.

Get Updates

Stay on top of what we’re working on by subscribing to our email digest.

optional

Our Hottest Stories

  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •  
  •