The Power of Small Data: Defend against data-doubting critics by exploring all angles

You strike a match and light a cigarette.

The cause of the smoke is clear and not up for debate.

The link, though, between that smoke and the lung cancer you may develop years later was the subject of debate for decades. And it was not until 40 years after the first warning labels were put on packs of cigarettes in the United States that scientists discovered what they declared a causal link.

Very few other risk factors for disease and early death are as well researched as smoking and, as a result, there are far fewer established links between, for example, one particular food and one particular outcome, good or bad.

Don’t let that scare you away from writing data-driven stories about health challenges and their possible solutions. Just be prepared for critics to say, “It’s not causation. It’s a correlation.” Here are three things you can do to strengthen your reporting:

1. Learn the terms. You might think that causation and correlation are totally different things. But you will be surprised when you start to probe the scientific literature and discover many things that we take for granted as being “caused by” something are usually somewhere on a spectrum that ranges from “indisputable truth” to “best guess.” Researchers not only write about “correlations,” but also about “associations,” “determinants,” “risk factors,” and many other terms. There is a whole literature, for example, dedicated to the concept of “causal inference,” including an entire journal and scientific conferences devoted to the topic. To get a sense of the depth of the discussion around how we can best assess whether a bad health outcome is caused by something specific, read Adrian Renton’s piece in the Journal of Epidemiology and Community Health on causal inference. He writes:

The consistent association between a factor and a disease occurring in correct time order in observational studies, where bias has been minimized, suggests a causal or confounded relationship. A strong relationship which persists in the face of strenuous attempts to control confounding in observational studies and through intervention studies shifts the balance towards causation. A knowledge of the mechanisms of pathogenesis of the disease, and the demonstration that a factor will materially influence these mechanisms through the material laws which govern them, adds further to our confidence in causation.

All of that just to get us to the point where we are starting to feel more confident in causation!

2. Do your homework. Try your best to understand the state of the evidence to date and what you are adding to that evidence. This will put you in a good position to explain why an association or correlation is quite strong — or quite weak.

That’s what reporters had to do with tobacco for years before a convincing link was found in 1996. As evidence piled up, reporters made note of that. Here’s how David Stout described the situation in The New York Times in 1996:

While many scientists have long been convinced by statistical studies and animal experiments that tobacco causes cancer, a statistical association was not in itself absolute proof. This shortfall has allowed defenders of smoking to deny that cigarettes cause cancer, and scientists have not known the exact mechanism of causation that would put the matter beyond doubt.

3. Don’t expect perfect data for anything, especially health topics. As I wrote earlier in this series, most of what you are seeing in the health literature, in government reports, and in news stories from people quoting the same are estimates that are based on the best available evidence. Those estimates are based on hosts of assumptions. Your job as a reporter is to try your best to explain the reasons for the claims being made in the story, the caveats for the claims, and the quality of the evidence in support of them.

Tena Rubio did a masterful job in her piece for KCRW about pollution from trucks at the Port of Los Angeles. The health effects of air pollution are as hotly contested as the link between tobacco and cancer once was. But the evidence that air pollution causes a range of bad health outcomes grows stronger every year as the research piles up. Rubio explored the different ways that pollution was having an impact in her piece and did not simply turn in a “he said/she said” story pitting the people breathing bad air against the American Trucking Association. By spending enough time with the topic, Rubio was able to explain to her audience what they needed to know to decide for themselves whether they should be concerned about pollution at the port.

[Photo by Pete via Flickr.]