Not all medical studies are equal: A physician-journalist explains how to evaluate scientific evidence

Dr. Seema Yasmin, a health officer for the Centers for Disease Control and Prevention (CDC), was investigating an outbreak of invasive Group A streptococcal infections, also called flesh-eating bacteria, on a Native American reservation, when she had a “journalism ah-ha moment.” She was interviewing a mother worried about protecting her young children. Yasmin told her that hand washing would keep her family safe, not realizing there was no running water.

Suddenly Yasmin understood: “If I’m going to be an effective public health doctor, I’m going to have to learn about communication and effective storytelling.”

Yasmin, now an author and a CNN medical analyst who worked for the Epidemic Intelligence Services of the CDC, shared her insights about health reporting last week with the USC Center for Health Journalism 2023 Health Equity Fellows. Bridging the disconnect between the scientists who generate and disseminate information and the people it impacts has been a driving force for her work.

Yasmin began her talk with her childhood, as it determined her path and continues to influence her approach to reporting.

“It’s important to share why we do what we do,” she said.

She was born in England into a conservative Indian Muslim family. Her mother chose to leave behind the traditional expectations for women in her culture to attend university. Yasmin was 5 years old, and when her mother went to class, university librarians and science documentaries became her babysitters. That’s when she developed the love of science that led her to medical school.

In a conversation with Jacqueline Stenson, project manager for the Center for Health Journalism, Yasmin offered tips for smart reporting on health and medical research.

Journalists should be as diligent about examining where information comes from as they are about verifying the information itself.

“I want to highlight not just fact checking, but also source checking,” she said.

PubMed is a good resource for finding research and experts on a topic, but it can be overwhelming. Yasmin said reporters can consider a journal’s impact factor, a score for how often the publication is cited by researchers. Well-respected journals such as Nature and The New England Journal of Medicine can also provide reliable sources.

Understand that not all scientific studies are equal.

She reviewed the hierarchy for quality of medical studies, using a pyramid for the strength of scientific evidence, with the top being the strongest. At the pinnacle are meta reviews, which involve the evaluation of multiple, even hundreds of studies providing data from different sources. Just below meta reviews are randomized placebo-controlled trials, which are usually double-blinded, meaning neither the participant nor the researcher knows who is getting treatment or a placebo.

Next are cohort studies and then case control studies. Both compare different groups but their designs differ. Cohort studies look at disease occurrence in exposed and non-exposed groups, for example rates of asthma among people living near a highway compared to those who don’t. Case control studies look at people with a disease (cases)  compared to people without it (controls). At the bottom of the pyramid are expert opinion pieces.

Know what you’re getting in pre-prints.

Preprints, early drafts of research reports, garnered a lot of attention in recent years, especially during the COVID-19 pandemic. These papers may appear on the websites of well-regarded journals or open-source websites, but they haven’t been peer-reviewed – independently analyzed by experts in the field. Peer review is the gold standard for scientific and academic research. “This gives it legitimacy and credibility,” Yasmin said.

Interrogate medical studies by asking these questions:

● What are the conflicts of interests for the study authors?

● How was the study funded?

● Where does the author’s department get funding?

● Which findings are statistically significant and which are not?

● What question was the study designed to answer?

● Do the results speak to that?

● What are the limitations of the study?

Yasmin also cautioned journalists not to rely solely on a press release, but to talk with the researchers who performed the study.

Stenson asked about Yasmin experiences combating misinformation, especially during COVID-19. , “It’s not a new problem,” Yasmin said. ”That’s the reason I wrote “What the Fact?!,” a book on the topic.

Sometimes, she said, how we talk about science can be alienating. She gave the example of the changing recommendations about the utility of masks early in the pandemic. Experts initially said they weren’t helpful, then changed the message and urged everyone to wear masks. Instead of understanding that science is a process and new evidence can lead to new recommendations, some people lost trust in medical experts amid the confusion and contradictory guidance.

“The long-term solution is to build science literacy,” said Yasmin.