Ratings built on rep

When U.S. News & World Report  released its annual rankings in July, Parkland Memorial Hospital was named one of the nation's best. It was the 18th straight year Parkland had won such recognition.

But just as the magazine released its rankings, government regulators began their second week of health and safety inspections at the Dallas County public hospital. 

The U.S. Centers for Medicare & Medicaid Services found that conditions there put patients in "immediate jeopardy." It threatened to revoke the hospital's Medicare and Medicaid funding until it accepted outside safety monitors. CMS has cited Parkland at least 14 times in the last five years for endangering or harming patients.

So how can a hospital be judged so deficient by federal inspectors, yet rank among the best in U.S. News & World Report ?

It's all in the methodology.

U.S. News  allows a hospital's reputation among doctors to account for nearly one third of its total score. The magazine says it's a form of peer review, but critics question its validity. 

The physician survey asks board-certified specialist physicians to name five outstanding hospitals in their speciality.

About 200 doctors are randomly selected for each specialty’s rankings. They are told not to factor geography or cost of care into their opinion. The surveys' results are combined with other data to determine rankings in 12 of 16 specialties. The other four specialties are ranked on reputation alone.

Avery Comarow, the magazine's health rankings editor, said he agreed that more hard statistical evidence would improve the ratings. Even so, he and his colleagues argue that a hospital's reputation among physicians says something important about its quality of care.

Feedback loop

More than a decade ago, researchers at New York University Medical Center called the rankings "little more than a feedback loop that allows fame to be perceived as quality."  

In April, Dr. Ashwini R. Sehgal, a Case Western Reserve University professor, reached a similar conclusion after studying the magazine's rankings. He determined that they "largely reflect the subjective reputations of those hospitals."

Sehgal compared the reputation rankings of the top hospitals in each of the 12 specialties to the hospitals' final scores once other data were combined. He found that for 89 percent of the hospitals in each top 20, the two lists were the same.

"The current rankings fall short of being an evidence based system that data-conscious consumers, value-based purchasers, and reform-minded policymakers can rely on for health care decisions," Sehgal wrote in the Annals of Internal Medicine .

Paul Levy, the former chief executive of Beth Israel Deaconess Medical Center, a HarvardUniversity teaching hospital, also said the opinion may not reflect reality.

"With great respect for the 200 pulmonologists who were surveyed, how much current data have they seen about the outcomes achieved by hundreds of hospitals and thousands of doctors around the country. Answer: None. Why? Because there is no current data published on such outcomes," Levy wrote on his medical blog in August. "Likewise, there is no current data published about hospital related infections, falls, medication errors and other matters that could affect the treatment of a pulmonary patient, even if the pulmonologists are top-notch."

The nation's leading hospital accreditation board, the Joint Commission, recently named 405 hospitals as being among the best, based on statistical evidence, for things such as heart attack and pneumonia. Missing were many hospitals, such as the Johns Hopkins Hospital and the Mayo Clinic, that consistently rank high inU.S. News & World Report.

Starting point

The magazine began publishing its hospital rankings in 1990 and says they are the "most comprehensive annual ranking of the nation's top medical centers."

"The purpose from the beginning has been to give consumers a starting point that they can consider useful in finding a hospital that's the best for them," said Comarow.

The magazine gives equal weight — 32.5 percent — to the opinion surveys and to hard data, such as Medicare claims showing how well a hospital kept its sickest patients alive. 

An additional 30 percent of a hospital's score in each specialty is determined by factors such as accreditation, nurse staffing levels, technology and other quality-of-care measures, mainly as self-reported to theAmerican Hospital Association. 

Since 2009, 5 percent of a hospital's score has come from its performance on a patient safety indicators index, similar to the one used by The Dallas Morning News . Comarow says the percentage is low because "it isn't clear that all hospitals are equally diligent and accurate in submitting PSI-related information to the government."

He added that the magazine's rankings are not a top-tobottom review of a hospital's performance. Hospitals in each metro area are ranked by the number of specialties in which they scored highly. 

"In those specialties, we are looking at a tiny little subset of patients," Comarow said. "We're not looking at what happens in the ER. We're not looking at what happens with routine care. There are lots of things we aren't looking at."