Editor’s Note: As we reported last
month, preventable harm in hospitals is now the third-leading
cause of death in the U.S. That makes it more important than ever to know if
your hospital is doing a good job. But can patients trust the many web sites
that rate hospitals? ProPublica’s Charles Ornstein
compiled the post below to help journalists interpret the ratings, but the
advice is just as critical for health care consumers. The bottom line: Use the
ratings with caution.

Few things in health journalism make me
cringe more than news releases touting hospital ratings and awards. They’re
everywhere. Along with the traditional U.S.
News & World Report rankings, we now
have scores and ratings from the Leapfrog Group,
Consumers Union,
HealthGrades, etc.

I typically urge
reporters to avoid writing about them if they can. If their editors mandate it,
I suggest they focus on data released by their state health department or on
the federal Hospital Compare website. I also tell reporters to be sure to check
whether a hospital has had recent violations/deficiencies identified during
government inspections. That’s easy to do on the website hospitalinspections.org, run by the Association of Health Care Journalists.
(Disclosure: I was a driving force behind the site.)

Last week, I got an
email from Cindy Uken, a diligent health reporter from the Billings (Mont.)
Gazette. She was seeking my thoughts on covering hospital ratings. I sent her a story written by Jordan Rau of Kaiser Health News about the proliferation of
ratings. Two
of every three hospitals in Washington, D.C., Rau
reported, had won an award of some kind from a major rating group or company.
He pointed out how hospitals that were best-in-class in one award program were
sometimes rated poorly in another.

This got me thinking: What should
reporters tell their editors about hospital rankings, ratings and awards?

I sought advice from Rau, along with ProPublica’s Marshall Allen, Steve Sternberg of U.S. News
& World Report, and John Santa of Consumers Union. Here’s what they told
me:

* * *

Steve Sternberg, Deputy Health
Rankings Editor at U.S. News & World Report

Reporters should cover hospital
rankings and ratings, going deeper than the knee-jerk stories you often see
suggesting that proliferating rankings and ratings confuse consumers. Hospital
rankings and ratings shouldn’t be expected to tell one story; they provide
different perspectives on hospital care. U.S. News Best Hospitals and Best
Children’s Hospital rankings evaluate the extent to which hospitals can provide
sophisticated care for really sick adults and children. We’re now in the early
stages of figuring out how to evaluate how well U.S. hospitals perform routine
procedures for patients who need routine care. Other rankings and ratings
evaluate other dimensions of care, such as patient safety.

This is fertile ground for reporting
and an important public service. Most people wouldn’t think of buying a house
without checking into the sale price of the houses next door, local transportation,
the quality of local schools and shopping nearby. Many don’t realize that they
can also check into the performance of the local hospital, starting with the
hospital’s website, the government’s Hospital Compare
website and other data, including the rich hospital data on the U.S. News
website. Rankings and ratings can provide useful information, as long as you do
your homework, recognize their limitations and, of course, talk to your doctor.
Good reporting can guide people to the information they need to make critical
healthcare choices.

* * *

Marshall Allen, Reporter,
ProPublica

Reporters should be cautious about
hospital rating methods for many reasons: They use different types of data,
different collection methods, different quality measures, and
indexing/weighting/aggregating of the data that is presented in different ways.
So it’s hard to draw conclusions that are too broad because there are so many
different variables in play. 

For example, Medicare measures what are
called Hospital Acquired Conditions (certain type of infections, falls that result in injury,
etc.), which are derived from billing records, which can be inconsistent in the
way they are coded. That inconsistent coding could result in some hospitals
appearing to have more HACs than others. But the differences in these numbers
may just show which hospitals more carefully and accurately code for HACs, not
which hospitals actually have more patients suffer from them. Making this worse
is the fact that the hospitals that most carefully and honestly code the HACs
would appear to have the worst rates, whereas hospitals that just don’t code
them would appear to have the highest levels of quality.

Medicare obviously uses the HAC data,
and the Leapfrog Group also makes them part of its Patient Safety Score
measure. Personally, I think that when HACs are coded they are likely to have
happened, but the inconsistency in coding makes them something I would be wary
about using for any comparison between hospitals. 

The U.S. News & World Report
rankings depend in part on how other doctors feel about a hospital. But this
reputation measure is determined by U.S. News asking doctors to write down the
hospitals that they think are best for each specialty. 

It’s very important for reporters to
read the methodology of each scoring system. You have to understand the source
of each type of data that makes up a measure, how it’s gathered, processed and
presented. That’s the only way to really understand each quality measure.

* * *

John Santa, M.D., Director,
Consumer Reports Health Ratings Center

There’s a lot happening with hospital
report cards. There’s more and better information especially around safety;
more and maybe better organizations wading in. Some raters are challenging
hospitals in a variety of ways — i.e., if you disagree with the rating,
make the data that supports your criticism public.

It is important to inform readers about
the bias publishers of report cards may have. I recently saw a piece where the
writer was surprised to find that HealthGrades does
consulting work for hospitals, including many of the hospitals it grades well.
Likewise, many folks do not connect the dots and realize that U.S. News
supports its hospital ratings efforts in part by selling the ratings to
hospitals to use in their ads.

Realize the most powerful “raters” are
hospitals themselves. They are spending hundreds of millions of dollars on ads
which often “rate” themselves — their ads use terms like “best,” “most
advanced,” “leading,” usually with little if any transparent validation. I
recently saw an ad in Southwest Airlines magazine from University of Texas
Medical Branch in Galveston that said their 30 minimally invasive surgeons were
the “best” in the country. Really — where did that data come from??
Amazing that every one of their surgeons is in the best group. Seems unlikely.
But this stuff makes a difference because consumers see it over and over again.

(Sternberg later wrote me to say, “Hospitals that perform well may license our logo
for advertising or promotional purposes. In no way, however, do we ‘sell’ the
rankings to anybody or allow hospitals to influence our results.”)

* * *

Jordan Rau, Kaiser Health News

Many of the judgments these groups make
about a hospital — whether to put them on a top 10 list or give them an A
or a B — are based on statistically insignificant comparisons. In the Joint Commission’s
rankings of top performers, 583 hospitals missed out on making that list
because they fell short on just one of 45 different measures. Often the
difference between a ranking of two hospitals is just 1 percent or 2 percent,
and the one that’s higher gets a better grade or makes it on a best hospital
list even though that difference doesn’t matter.

That’s not just the case with private
report cards. On Medicare’s Hospital Compare
website, measures of patient satisfaction, timely and effective care, medical
imaging use and hospital acquired conditions are not presented in a way in
which you can tell where a difference between two hospitals — or from the
national or state rate — is significant or not. Hospital Compare data is
a common source for most of these private ratings, so that becomes a problem as
they are aggregated into composite measures like grades or list rankings.

As a rule of thumb, reporters should be
wary if you are presented with two numbers and no way to know at what point the
difference matters. Think of it like a public opinion poll: Until you know the
margin of error, you don’t know if the results matter.

Another reason to be
skeptical is that, as we noted in one of our stories, “Hospital
Ratings Are In The Eye Of the Beholder”,
the economic models behind most of these report cards provide little incentive
to downplay differences. These rating groups actually have a financial
incentive to be liberal in handing out accolades, because they often make their
money by licensing their awards or ratings to the hospitals that get the nod.
It’s hard to sell an analysis that says 90 percent of hospitals are average.

A broader problem is that these report
cards are mostly measuring a small subset of procedures and conditions, so
reporters need to be careful about extrapolating from them to a hospital’s
overall quality. If a hospital does well or poorly on pneumonia, heart attack
and heart failure patients — the group that is most commonly measured —
that doesn’t necessarily mean they do a great job with knee surgery or a heart
transplant. What often matters there is the surgeon and team, not the hospital.

It’s probably safer to extrapolate
about things like hospital acquired infections, because those reflect the
safety culture of a hospital, but as Marshall notes, the places that are more
faithful in reporting problems can look worse than those that are cavalier
about reporting. A good maxim to keep in mind is this one from (sociologist) William
Bruce Cameron: “Not everything that can be counted counts. Not everything that
counts can be counted.”

All that said, the
measures that identify hospitals that are statistically better or worse —
mortality is probably the best example — are worth using. And I think
it’s fine to use a rating as one piece of a broader assessment, one that might
include regulatory fines or reports or evidence in lawsuits.