Editor’s Note: As we reported last month, preventable harm in hospitals is now the third-leading cause of death in the U.S. That makes it more important than ever to know if your hospital is doing a good job. But can patients trust the many web sites that rate hospitals? ProPublica’s Charles Ornstein compiled the post below to help journalists interpret the ratings, but the advice is just as critical for health care consumers. The bottom line: Use the ratings with caution.

Few things in health journalism make me cringe more than news releases touting hospital ratings and awards. They’re everywhere. Along with the traditional U.S. News & World Report rankings, we now have scores and ratings from the Leapfrog Group, Consumers Union, HealthGrades, etc.

I typically urge reporters to avoid writing about them if they can. If their editors mandate it, I suggest they focus on data released by their state health department or on the federal Hospital Compare website. I also tell reporters to be sure to check whether a hospital has had recent violations/deficiencies identified during government inspections. That’s easy to do on the website hospitalinspections.org, run by the Association of Health Care Journalists. (Disclosure: I was a driving force behind the site.)

Last week, I got an email from Cindy Uken, a diligent health reporter from the Billings (Mont.) Gazette. She was seeking my thoughts on covering hospital ratings. I sent her a story written by Jordan Rau of Kaiser Health News about the proliferation of ratings. Two of every three hospitals in Washington, D.C., Rau reported, had won an award of some kind from a major rating group or company. He pointed out how hospitals that were best-in-class in one award program were sometimes rated poorly in another.

This got me thinking: What should reporters tell their editors about hospital rankings, ratings and awards?

I sought advice from Rau, along with ProPublica’s Marshall Allen, Steve Sternberg of U.S. News & World Report, and John Santa of Consumers Union. Here’s what they told me:

* * *

Steve Sternberg, Deputy Health Rankings Editor at U.S. News & World Report

Reporters should cover hospital rankings and ratings, going deeper than the knee-jerk stories you often see suggesting that proliferating rankings and ratings confuse consumers. Hospital rankings and ratings shouldn’t be expected to tell one story; they provide different perspectives on hospital care. U.S. News Best Hospitals and Best Children’s Hospital rankings evaluate the extent to which hospitals can provide sophisticated care for really sick adults and children. We’re now in the early stages of figuring out how to evaluate how well U.S. hospitals perform routine procedures for patients who need routine care. Other rankings and ratings evaluate other dimensions of care, such as patient safety.

This is fertile ground for reporting and an important public service. Most people wouldn’t think of buying a house without checking into the sale price of the houses next door, local transportation, the quality of local schools and shopping nearby. Many don’t realize that they can also check into the performance of the local hospital, starting with the hospital’s website, the government’s Hospital Compare website and other data, including the rich hospital data on the U.S. News website. Rankings and ratings can provide useful information, as long as you do your homework, recognize their limitations and, of course, talk to your doctor. Good reporting can guide people to the information they need to make critical healthcare choices.

* * *

Marshall Allen, Reporter, ProPublica

Reporters should be cautious about hospital rating methods for many reasons: They use different types of data, different collection methods, different quality measures, and indexing/weighting/aggregating of the data that is presented in different ways. So it’s hard to draw conclusions that are too broad because there are so many different variables in play. 

For example, Medicare measures what are called Hospital Acquired Conditions (certain type of infections, falls that result in injury, etc.), which are derived from billing records, which can be inconsistent in the way they are coded. That inconsistent coding could result in some hospitals appearing to have more HACs than others. But the differences in these numbers may just show which hospitals more carefully and accurately code for HACs, not which hospitals actually have more patients suffer from them. Making this worse is the fact that the hospitals that most carefully and honestly code the HACs would appear to have the worst rates, whereas hospitals that just don’t code them would appear to have the highest levels of quality.

Medicare obviously uses the HAC data, and the Leapfrog Group also makes them part of its Patient Safety Score measure. Personally, I think that when HACs are coded they are likely to have happened, but the inconsistency in coding makes them something I would be wary about using for any comparison between hospitals. 

The U.S. News & World Report rankings depend in part on how other doctors feel about a hospital. But this reputation measure is determined by U.S. News asking doctors to write down the hospitals that they think are best for each specialty. 

It’s very important for reporters to read the methodology of each scoring system. You have to understand the source of each type of data that makes up a measure, how it’s gathered, processed and presented. That’s the only way to really understand each quality measure.

* * *

John Santa, M.D., Director, Consumer Reports Health Ratings Center

There’s a lot happening with hospital report cards. There’s more and better information especially around safety; more and maybe better organizations wading in. Some raters are challenging hospitals in a variety of ways — i.e., if you disagree with the rating, make the data that supports your criticism public.

It is important to inform readers about the bias publishers of report cards may have. I recently saw a piece where the writer was surprised to find that HealthGrades does consulting work for hospitals, including many of the hospitals it grades well. Likewise, many folks do not connect the dots and realize that U.S. News supports its hospital ratings efforts in part by selling the ratings to hospitals to use in their ads.

Realize the most powerful “raters” are hospitals themselves. They are spending hundreds of millions of dollars on ads which often “rate” themselves — their ads use terms like “best,” “most advanced,” “leading,” usually with little if any transparent validation. I recently saw an ad in Southwest Airlines magazine from University of Texas Medical Branch in Galveston that said their 30 minimally invasive surgeons were the “best” in the country. Really — where did that data come from?? Amazing that every one of their surgeons is in the best group. Seems unlikely. But this stuff makes a difference because consumers see it over and over again.

(Sternberg later wrote me to say, “Hospitals that perform well may license our logo for advertising or promotional purposes. In no way, however, do we ‘sell’ the rankings to anybody or allow hospitals to influence our results.”)

* * *

Jordan Rau, Kaiser Health News

Many of the judgments these groups make about a hospital — whether to put them on a top 10 list or give them an A or a B — are based on statistically insignificant comparisons. In the Joint Commission’s rankings of top performers, 583 hospitals missed out on making that list because they fell short on just one of 45 different measures. Often the difference between a ranking of two hospitals is just 1 percent or 2 percent, and the one that’s higher gets a better grade or makes it on a best hospital list even though that difference doesn’t matter.

That’s not just the case with private report cards. On Medicare’s Hospital Compare website, measures of patient satisfaction, timely and effective care, medical imaging use and hospital acquired conditions are not presented in a way in which you can tell where a difference between two hospitals — or from the national or state rate — is significant or not. Hospital Compare data is a common source for most of these private ratings, so that becomes a problem as they are aggregated into composite measures like grades or list rankings.

As a rule of thumb, reporters should be wary if you are presented with two numbers and no way to know at what point the difference matters. Think of it like a public opinion poll: Until you know the margin of error, you don’t know if the results matter.

Another reason to be skeptical is that, as we noted in one of our stories, “Hospital Ratings Are In The Eye Of the Beholder”, the economic models behind most of these report cards provide little incentive to downplay differences. These rating groups actually have a financial incentive to be liberal in handing out accolades, because they often make their money by licensing their awards or ratings to the hospitals that get the nod. It’s hard to sell an analysis that says 90 percent of hospitals are average.

A broader problem is that these report cards are mostly measuring a small subset of procedures and conditions, so reporters need to be careful about extrapolating from them to a hospital’s overall quality. If a hospital does well or poorly on pneumonia, heart attack and heart failure patients — the group that is most commonly measured — that doesn’t necessarily mean they do a great job with knee surgery or a heart transplant. What often matters there is the surgeon and team, not the hospital.

It’s probably safer to extrapolate about things like hospital acquired infections, because those reflect the safety culture of a hospital, but as Marshall notes, the places that are more faithful in reporting problems can look worse than those that are cavalier about reporting. A good maxim to keep in mind is this one from (sociologist) William Bruce Cameron: “Not everything that can be counted counts. Not everything that counts can be counted.”

All that said, the measures that identify hospitals that are statistically better or worse — mortality is probably the best example — are worth using. And I think it’s fine to use a rating as one piece of a broader assessment, one that might include regulatory fines or reports or evidence in lawsuits.