How to illuminate the strengths and limits of data in your stories

This is the latest post in an ongoing series called “The Power of Small Data." You can find earlier installments here. — Ed.

Face validity.

Spend any time in scientific circles, and you’re likely to hear this term come up, usually as part of a critique. “That simply doesn’t pass face validity.”

When Nick Miller was with the Sacramento News & Review, Sacramento County published a survey purporting to capture the number of children who were homeless. Out of a county of more than 350,000 children — by most estimates — the survey found just five kids who were homeless. Miller had to figure this didn’t pass face validity.

But he didn’t just go with his gut. He went and talked with people who were closer to the homeless situation in the county and found stronger evidence to compare against this weak evidence. It’s not that the survey that concluded that 0.001% of the youth population in the county were homeless was worthless. It’s just that it probably didn’t do the best job capturing the true extent of homelessness in that age group. Miller wrote (emphasis mine):

But the proverbial boots-on-the-ground workers … say the county data is flat-out wrong. Consider a report by county school districts from last year. The study had teachers from 13 area school districts ask students about their living situations. What they learned: 11,354 Sacramento County kids were in homeless situations, the most being 964 first graders.

Miller explained nicely where the data came from and gave his audience enough information — without having to write a scientific paper — for them to judge whether to put more stock in the county’s survey or the survey from the school districts.

You can do the same thing with your own data. Don’t be afraid to show its strengths and the weaknesses. How did you gather your data? What sort of process did you use to make any estimates that you are making? Where might your efforts have fallen short?

This last question I find particularly hard for reporters to address, largely because it’s hard enough to persuade an editor to let you spend time compiling data. Why would you want to expose the problems inherent in the data? At a minimum, you should explain what’s included in your findings and what is not included. If there are real limitations to your data, address them.

The Washington Post has done an incredible public service with its extensive efforts to document police shootings in the U.S. And it also is trying to be very open about those efforts. Some of the Post staff involved in the project wrote an explanatory piece called “How The Washington Post is examining police shootings in the U.S.” They write:

The Post is documenting only those shootings in which a police officer, in the line of duty, shot and killed a civilian — the circumstances that most closely parallel the 2014 killing of Michael Brown in Ferguson, Mo., which began the protest movement culminating in Black Lives Matter and an increased focus on police accountability nationwide. The Post is not tracking deaths of people in police custody, fatal shootings by off-duty officers or non-shooting deaths.

I have no idea when the first journalist wrote a “How I did this story” box to accompany their piece, but it certainly became more common in the 1990s and continues today, sometimes with the stories about the data being just as interesting as the stories themselves (at least if you’re a reporter).

Tell your audience what you did and why it matters. Also tell your audience enough about the evidence for them to make a reasonable assessment and, even better, to ask their own questions that take the story even further.

—

Related posts

The Power of Small Data: Illuminate data absolutely and relatively, too

The Power of Small Data: Why you probably don’t need ‘big data’ for your stories

The Power of Small Data: Lessons learned from a number-crunching career

The Power of Small Data: When Stymied, Make Your Own Database

The Power Of Small Data: Doubt Those Who Say They Have No Numbers

The Power of Small Data: Brace yourself for the data-doubters

The Power of Small Data: Ask Questions That Demand Answers

The Power Of Small Data: Defend Against Data-Doubting Critics By Exploring All Angles

The Power of Small Data: Weigh the evidence before you report it

The Power of Small Data: How Rwanda tried to save lives with better math

[Photo by Sebastiaan ter Burg via Flickr.]