ProPublica

Journalism in the Public Interest

Cancel

Why Do Hospital Generators Keep Failing?

The power failure at New York University Langone Medical Center during Hurricane Sandy shows that hospitals still may not be doing enough to prepare for disasters.

Hospital workers evacuate a patient from NYU Langone Medical Center during Hurricane Sandy on Oct. 29, in New York City. (Michael Heiman/Getty Images)

It is a hospital's nightmare: The power goes out and backup generators don't kick in, leaving critically ill patients without the mechanical help they need to breathe.

It happened in New Orleans after Hurricane Katrina, when hospital staff were on their own when electricity and water cut out. Some died.

It happened last year in San Diego, when generators at two hospitals failed during a blackout.

And it happened last year in Connecticut, when a hospital had to be evacuated during Hurricane Irene when its generator failed.

As Hurricane Sandy headed toward the East Coast, New York City Mayor Mike Bloomberg said Sunday, he was assured that hospitals were ready. "The teams from the City Health Department are at these facilities making sure that the emergency generators are working and that they have back-up fuel supplies," he said at a news conference before the storm.

In spite of this confidence, the generators at some hospitals did not work as expected. In the most high-profile case, New York University Langone Medical Center had to evacuate all 215 of its patients when its power went out and both of its backup systems didn't work. Staff had to hand pump oxygen to critically ill patients until patients could be taken by ambulance to another hospital.

In New Jersey, patients at Palisades Medical Center had to likewise be evacuated in recent days after two generators failed. A hospital spokesman said today that officials were working to bring the "plant up to speed" and could not immediately answer questions about the generators.

And this afternoon, Bellevue Hospital in New York City said it is evacuating hundreds of patients because of failing power and deteriorating conditions. "It's Katrina-esque in there," one nurse told ABC News.

Experts say such failures are troubling but not entirely surprising. Dr. Arthur Kellermann founded the emergency department at Emory University and headed it from 1999 to 2007. Now, he's Paul O'Neill-Alcoa Chair in Policy Analysis at RAND Corporation think tank.

The other night, as the NYU evacuation was unfolding, he tweeted, "Hospital preparedness and well-functioning backup systems are a costly distraction from daily business, until they are needed. Like now."

In an email interview with ProPublica, Kellermann elaborated: "I have no doubt when the hospital assured the Mayor that their backup systems were ready, they believed they were. They were wrong. What I find most remarkable about this story is that [more than seven] years after Hurricane Katrina, major hospitals still have critical backup systems like generators in basements that are prone to flooding."

Bruce Altevogt, a senior program officer at the Institute of Medicine who has studied crisis standards of care, lauded NYU for safely evacuating patients. But he said the incident should prompt a new discussion about where hospitals place generators and how to ensure they work when they're needed.

Newly constructed hospitals are supposed to place their generators and fuel in adjacent locations above flood level. But the location requirements do not apply to already-built hospitals.

"These older facilities, it's just an economic issue," Altevogt said. "They don't have the resources, or they haven't devoted the resources to moving the equipment to locations that would be less prone to disasters or flooding in these cases."

An NYU spokeswoman told Modern Healthcare magazine Tuesday that "part of the generator is on the roof and part in the basement, which she said took on 8 feet of water. The reason for the failure, she said, is being investigated."

New York hospitals have had experiences with generator failures before and have seen the consequences firsthand.

In 1987, a 22-minute power failure led to the death of a 40-day-old infant at New York Hospital. "The prematurely born infant, whom officials declined to identify, died after his electric respirator stopped and efforts to help him breathe with a manual air pump failed," the New York Times reported at the time.

A decade before that Bellevue Hospital lost all power during the massive New York City blackout in 1977. The Times reported at the time: Doctors and nurses squeezed bags of air with their hands to resuscitate patients when respirators stopped, and emergency generators were brought in."

During another New York City-wide blackout in 2003, an official report later said, "Despite prior testing according to applicable State and accreditation standards, [some] generators malfunctioned, experiencing, for example, problems with switches and overheating. ...In a few cases hospitals reported that fuel supplies for generators fell to dangerously low levels, in part because of transportation difficulties encountered by fuel delivery trucks."

Back at NYU, a hospital trustee, Gary Cohn, said that the board knew the facilities' generators were outdated and at risk, according to Bloomberg Businessweek.

"The infrastructure at NYU is somewhat old," Cohn said in a television interview. The backup generators "are not state-of-the-art and not in the most state-of-the-art location."

A hospital spokeswoman defended its systems to the Huffington Post.

"Our generators are fully compliant with all state and federal regulations and, using good prudence, we test them all the time as we have to do anyway," Lisa Greiner said.

Emails to NYU staff by ProPublica were not returned prior to publication.

Another expert in disaster planning, Dr. Dan Hanfling, said in an email that the failure of the back-up power at NYU Langone "is certainly unexpected."

Hospitals, he said, are required by the Joint Commission, a hospital accreditor, to have back-up power capabilities. They are required to "load test" those systems a few times each year. (Update 10/31: The Joint Commission says it requires tests 12 times a year for 30 minutes and once every three years for four hours.)   

"Evidently, the catastrophic failure must be explained by something else," wrote Hanfling, a special adviser for emergency preparedness and response for Inova Health System in Virginia.

In his email to ProPublica, Kellermann wrote that the irony of the NYU generator failure is that New York City's hospitals and health department "have taken preparedness more seriously than nearly everyone else in the country, particularly since 9/11/01." RAND tested a novel, no-notice disaster drill at a different city hospital a few months ago and it performed very well.

The situation at NYU should be a lesson for all.

"Preparedness is not simply a hoop to jump through to satisfy the Joint Commission, or to keep the Fire Marshall off your back," Kellermann wrote. "It is a fundamental duty to the community, state and country. Assuming nothing will ever happen, or counting on your staff to 'rise to the occasion' is not a plan, and it is not preparedness."

Hanfling went a step further. "Lesson to learn: hospitals must be prepared to evacuate patients. This may be the 'new normal' for hospital preparedness efforts," he wrote.

Why aren't hospitals better prepared for emergency events? Leave a comment, or join the discussion on Twitter with #InvestigateThis

This puts me in mind of the journalism of Greg Palast, who documented in “Vultures Picnic” that the backup generators in nuclear power plants are very fragile. They are designed after the model of ship power plants, huge pieces of tech which cannot be started quickly. So when a backup generator is put in service in the case of a nuclear emergency, it cracks the crankshaft in half. Don’t believe it? I admit it is unbelievable. Check the source yourself, I’m not here to do your research.

Backup generators and automated switches represent mature equipment. A lot of sites have them and they work very well. The problem is that the hospitals don’t consider the equipment key in their mission so they neglect them. They should be cycled and tested at least once a month on load. Not doing that is stupid if not criminal for hospitals. They check on all sorts of medical equipment and put stickers on them. I hope they are doing a better job at that than with the generators.

Our generators are even used by the local utility in a program that kicks in when there is high load on the grid such as high Summer. They can turn them on remotely. The problem with these at the hospitals is that they don’t take the systems seriously and then they crap out when they have an emergency.

We rarely learn from the mistakes of others.  In 2001 Tropical Storm Allison parked over Houston and did as much damage, or more, than several hurricanes before or since (stalled storm = tons of rain fall, over 24 inches in some parts of the city, couldn’t drive around the city for at least two days).  One of the major issues: backup generators in hospital basements.  More than one hospital had a high profile evacuation of patients carried on gurneys down stairs by staff - some hand pumping oxygen.  This wasn’t last year or two years ago - this was 11 years ago!

Generators that are just cycled occassionly and that are not run under a loaded condition can cause the Generator’s fuel valves to be what is called “Wet Stacking”. Where the valves stay saturated with fuel and after time dry and clogg or restrict the fuel injectors so when the Generator is required to run at full speed and under loaded condition The Generator may Fail.

Unfortunately, there’s no way to test a hospital generator thoroughly without basically ‘pulling the plug’ on incoming power.  The monthly and annual tests do the best they can, but they aren’t perfect. 

And if anyone really believes that these things should be perfect, the last perfect thing I heard of walked on water, and we aren’t it.  These are systems designed and run by humans - we all do the best we can and pray it’s enough. 

Many hospitals are running in the red, and moving generators up several floors in an existing hospital is a $1M+ project.  Add the need to remove patients from those floors during the construction and then add the disruption to the surrounding neighborhood during this process, well, it gets complicated and ugly, fast.

Haviing a workable electric backup system is not rocket science .. and when you have human lives at risk using terms like generators were compliant is a lame excuse when dealing with terminally sick people. Instead of spending billions of dollars on Obama care we should first make sure we can keep the lights on in our hopitals.. We seem to be regressing to a 3rd world standard of compliancy.. Wake up you pin heads..

Even more recently in 2011, we saw how critical cooling water pumps at Fukishima failed because they were located below the flood water line.

Basements are not ideal locations for any backup equipment.
Actually having them work when needed is another aspect…especially if rarely used for long periods of time.

Wonder how much it costs for a major metropolitan hospital to close and be forced to evacuate numerous patients vs the cost of relocating critical backup equipment. Which bottom line is being scrutinized?

maurice j. cassidy

Nov. 1, 2012, 12:52 p.m.

My company used to make custom fuel tanks for generators and the company we made them for begged me to make sure there was no welding slag or grit in the tanks. We had viewing ports to blow out the residue on the tanks and also vacume them clean. I was told that when a new generator was run the filters should be replaced every so many hours until they were sure the tanks were clean.  Most tanks are mass produced and the cleaning is not complete.  If they are not clean when they have an emergency they will only run until the filter clogs up and then stop running.  Clean tanks are a must

Another equally life-critical hospital infrastructure system that is often take for granted is the medical gas system.  The systems provide piped medical gases - Oxygen, Medical Grade Air, Nitrous Oxide and Surgical Vacuum, and are supposed to be maintained and inspected to rigorous standards since these gases are directly supplied to patients.  They are often misunderstood and neglected by hospital physical plant departments.

Dorothy Danaher White

Nov. 1, 2012, 1:16 p.m.

I have a doctorate in education from Harvard and I grew up in Miami. I’ve survived many hurricanes. I founded a non profit named Be Ready Bear (copyright 2008) which uses a friendly cartoon bear to educate adults and children about disaster preparedness. Educating about disaster is harder than educating about anything else. Racism, gender equity, homophobia, smoking, alcoholism - I’ve never before encountered more resistance! I need more help with this project. Be Ready Bear puppets can be dressed in scrubs and tend a toy generator. Believe me, its the most effective approach!

william readling

Nov. 1, 2012, 1:30 p.m.

I hold bachelors degrees in electrical and mechanical engineering.  Generators must be started, and run for a half hour or so, every month or so, not only to access their functionality, but to extend their lifespans.  The periodic operations lubricates the engine, dries out the lubricating oil, cleans old fuel out of the fuel system, dries the externals of the equipment, turns the bearings, charges batteries.  ect.  I’d also recommend they run the engine an once a week, until the oil temp got to operating temperatures, since operation is so critical.  It should go without saying, there should be an electrical load applied, so that that part of the system is tested too.
  These hospitals have the engine-alternator set anyway, that is the bit expense.  Having someone in maintenance, start, and run the equipment, and the cost of the few gallons of fuel is small.  In fact, doing this lowers the cost of maintenance, since it will decrease the numbers of repairs.  Imagine the costs of the wrongful death suits.

@ Mike W

That would explain why there has NEVER been a incident at a nuclear power plant caused by a failure of its diesel backups during a loss of offsite power event.

Allen Schaeffer

Nov. 1, 2012, 4:07 p.m.

There are thousands of emergency diesel generators operating today in the affected region and around the county at hospitals and other critical facilities.  You won’t hear about those because these units are working exactly as intended. 

However, as this story points out, there can be failures. and there is an important distinction to be made here in the notion of “generator failure.”

From what we’re seeing and hearing from the NY Region– most reported failures in NY/NJ are due to flooding of the generator rooms—as opposed to the mechanical or system failure of the units to start and operate properly (the other kind of failure). 

In this regard, Dr. Kellermann’s comment in this story was spot on.

The other situation that is also common in the high-rise world of NYC- the primary fuel supply and associated pumps are located at ground level/below grade and the generators are located at rooftop or higher floor, and the fuel supply is pumped to the generators.

When the main fuel supply room and the associated pumps go underwater, the generator units on the roof will not be able to function long before their smaller supply tank is depleted

Other commenters generally have it right here—exercising the units on a regular weekly basis, along with proper servicing and test operating under load conditions a few times a year—assures reaediness.
Design and specification of units, their location and assuring necessary redundancy and of couse good maintenance all contribute to systems doing their job under any cicumstances. 

Those are no doubt issues we’ll be hearing more about in days to come.

http://www.dieselforum.org

Robert Watson

Nov. 1, 2012, 6:23 p.m.

Another issue during and after Katrina is the depletion of on-site fuel stores. Working in the boiler industry, I see the lack of adequate fuel supplies at all the hotels and hospitals. None of these have more than two or at most three day supply of fuel. One university has a gas turbine co-gen unit which can carry most of the campus—but it burns five gallons a minute, 300 gallons an hour, 7200 gallons every 24 hours!! With a 60 hour reserve this one place needs at least 14,000 gallons every other day to keep going. How is this much fuel to be delivered in the middle of an on going crisis?? It can’t. Many places in New Orleans still have their engineering areas 35 feet below sea level!!! One place we work has been flooded “Katrina like” twice due to broken water mains in the street.

Gio Wiederhold

Nov. 2, 2012, 2:57 a.m.

Just testing systems when the weather is nice etc. is not adequate preparation for emergencies
There needs to be analysis for every major component in a system, and what it can be affected by.
Yes the fuel tanks, the fail-over switches (They were in the basement ay NYU?), proximity to roads and loading docks. Etc.
Failure analysis is a profession, but rarely a requirement for builders and contractors.
Without explicit requirements the lowest bidder wins.

Chris Diamond

Nov. 2, 2012, 7:32 a.m.

There is an old tale of an emergency generator, in a basement, well protected against flooding, with the electrical switchgear also well protected. When the power failed, the generator kicked in immediately, with power supplied to all essential circuits. Unfortunately someone decided that the sump pump was not essential. It cut out and the inevitable leaks gradually flooded the generator. Moral: stupidity trumps everything.

If the generators and their maintenance complied with the regulations, the regulations were inadequate. Why were the regulations not changed? Grandfathering to allow inadequate installations to remain is no solution (although modifying installations to safeguard the generators and switchgear might be enough). And what happened to “good engineering practice”?

See, there’s a conceptual problem, here:  A generator, like anything, is something you only notice—only know if it’ll do the job—seconds after you need it.

Yes, tests are great at picking up some issues, and I don’t know (and probably won’t learn) what happened to these generators.  But still, no test (as Paul points out above) can ever show the performance under the required, persistent demand except to run the backup live as the primary system, which would be a huge waste and probably end up endangering patients routinely.

@Mike H
Sorry, you’ve not done your homework. Check the source I mentioned. Why on earth would you make a claim like this with zero evidence to support it?

Hospitals will test the generators as soon as they can find someone to bill for the time.

I don’t think people know what actually happened here? I keep reading alot of stuff about generators in the basement and such. NYU Langone had all of it’s systems on the roof(generators, transfer switches, distribution switchgear). this was done several years ago. Although one thing that needs to happen is the main fuel tanks have to be on the ground floor, no other way to store large quantities of fuel. The fuel then goes through pumps up to the roof and stored in day tanks which provide a minimum 2 hour run time per generator. the fuel pump room is in a flood resistant room in the basement. When the forcefull flood of water entered the basement it made it’s way into the fuel pump room and tripped a safety feature meant to shut off the fuel pumps in case of a fuel spill. the pumps shut off and stopped delivering fuel at this point. the generators ran until they no longer had fuel in the day tank and shut off. by the time maintanance staff made there way to the fuel pump room to reset the fuel pump controller, they had already made the decision to evacuate the hospital.

There are three other caveats for hospitals to ponder; 1) everyone thinks their special piece of equipment is vital and essential, and they will find ways to connect it to the emergency power circuit. When the crunch comes and the emergency generator tries to spin up, the number of KWhr may be so much more than the designed operating specs when the system was put in (which likely pre-dated the PC explosion) that either the generator fails of the time-to-full-power is stretched out to the point where connected gear operates abnormally. 2) every piece of new hospital equipment that requires electricity to work should be vetted prior to purchase to determine whether transient loss of power results in return to the settings prior to power loss, return to function using “default” settings or return to idle/awaiting setting input. Surprisingly different manufacturers of medical gear have not adopted a common convention on this. Finally 3) if your institution’s engineering staff contracts out periodic maintenance of the emergency generators they need to check whether the generators are left in “auto-on-with-line power loss” mode, “require manual start if line power is lost”, or “off”. I personally had to assist the dialysis staff return the blood of two dialysis patients using cellphone backlighting because the Engineering staff at our hospital failed to verify that the emergency generators were left in “auto-on” after PM.

Load testing for prolonged periods of time is the best way to insure that these systems will actually work in a real emergency. Case in point: I had a facility that had the generator on the OR installed and tested as required over a 7 year period. Then there was an actual power interruption after a tornado. After about 5 hours of run time a low fuel alarm occurred. We found that the transfer pump from the main fuel tank to the belly tank on the generator set was not connected to an emergency power source. In the past when testing occurred the fuel used was replaced after the generator test period when city power was restored. The electrical contractor wired the unit according to the plans. There was an error on the plans. We were able to temporarily correct the problem before we ran out of fuel. We promptly corrected this once power was restored.

Imagine if Sandy happened when the NYU medical center prepared and cooking the food for 5 NYC hospitals including NYU medical center, Mount Sanai Hospital, NYU hospital for joint diseases, NYU downtown and NYU queens located in Astoria. 19500 meals a day
Who will do the work today to feed the patients of the oders hospital?
more centralisation is good in economics books not in reality and especialy in crisis

I see a lot of comments about load testing.  But if the NYU generators failed because of salt water infiltration into the fuel supply room how would this have been tested for?

This article is part of an ongoing investigation:
Patient Safety

Patient Safety: Exploring Quality of Care in the U.S.

More than 1 million patients suffer harm each year while being treated in the U.S. health care system. Even more receive substandard care or costly overtreatment.

The Story So Far

Too many patients suffer harm instead of healing in U.S. medicine. That’s why ProPublica’s reporters have investigated everything from deadly dialysis centers and dangerous hospitals to the failure of state boards to discipline incompetent nurses.


This page allows patients, providers and readers to join the patient safety conversation. Our goal is to find out why so many patients are suffering harm and highlight the best ways to solve the problem. Here you’ll find regular updates, and places to share your stories, views or expertise.

Read all of our posts on patient safety, and find out how to get involved.

Contact Us

Blair Hickman, Community Manager
@amandablair
Blair.Hickman@propublica.org

Got a tip? Fill out our form.

Share Your Story

Your input can help ProPublica's reporting.

Have you worked in health care? Tell us what you’ve observed about patient safety.

Have you or a loved one been harmed? Tell us about it.

Join the Discussion

Join the over 1,500 members of ProPublica's Patient Harm Group to learn, share your story and connect with others.

Icon graphics courtesy of the Noun Project.

Get Updates

Stay on top of what we’re working on by subscribing to our email digest.

optional