Independent Expert Helps Put Research In Perspective

In writing on early-childhood issues for this blog, one of the recurring themes I’ve touched upon is the idea that disparities – whether health, academic, or economic – often have roots very early in a child’s life, sometimes even before they’re born. The research has clearly shown that poverty and the chronic stress tends to put kids from poorer families at a big disadvantage from their better-off peers. And the gaps, left unheeded, can play out over a lifetime, shaping academic success, income, health and well-being, even lifespan. That’s why many early childhood researchers and policymakers are so eager to unravel these achievement gaps as early as possible.

So when a recent study published in the prestigious journal Proceedings of the National Academy of Sciences came along claiming that an eight-week “intervention” can significantly lower the academic achievement gap between classes, my curiosity was piqued. The temptation was to do a quick write-up on the study, add some context about how the findings bolster an emerging consensus on the importance of parental skill-building, and, oh yeah, maybe have an independent expert or two briefly comment on the validity of the research.

Except that as any reporter worth their beat will tell you, that last step is really the first and most important step. Granted, getting comments from independent experts can be a pain: They have busy lives, they often don’t return your emails or calls, they’ve never heard of your name or publication, and besides, voicing honest criticisms about their colleagues’ work in the press can complicate their own professional lives in all the expected ways. Yet with enough persistence, you can always get a qualified voice to talk about new work in a field they’re deeply invested in. Forgo those voices and it’s all too easy to end up with an over-credulous summary of research that may be far less robust than a hastily scanned “Conclusions” paragraph in the study suggests.

Study: Parent-Focused Approach Most Effective

The recent study from the Brain Development Lab at the University of Oregon seems to lend support for the idea that building low-income parents’ skills is a more effective way of boosting at-risk children’s abilities than focusing on just the kids alone. The eight-week study took 141 preschoolers enrolled in Head Start (the program targets low-income kids) and their parents and randomly assigned them to three different groups.

One group received Head Start instruction alone; a second group of children completed attention-building exercises (“selective attention” is key to academic success and correlated with kids’ socioeconomic backgrounds) and minimal parent training; and the third group consisted of kids who underwent attention-building activities but did so while their parents received weekly two-hour group classes covering strategies on topics such as regulating family stress, appropriate disciplining, language use in the home (more is better), and how to be a responsive parent. The study, building on past research, wanted to test the idea that parent training can improve low-income kids’ focus and academic abilities beyond what child-based exercises can do alone.

The results?

“The more parent-focused program was the clear winner,” said the study’s leader, Helen Neville, professor of neuroscience of the University of Oregon, Eugene, in a statement. The group in which both the kids and parents underwent the comprehensive 8-week training did better on language and cognition tests, and the parents reported that things had changed for the better for both their kids and themselves (significantly more so than in the other two control groups).

The study concludes with a big-picture thought:

The evidence presented here suggests that programs that target multiple pathways, including parents and the home environment, have the potential to narrow the large and growing gap in school readiness and academic achievement between higher and lower (socioeconomic status) children.

Expert: New Study ‘Essentially Trivial’

The researchers’ conclusion sounded like cause for optimism – better your family in only eight weeks! At least until I spoke with Dr. Glenn Flores, the Judith and Charles Ginsburg Chair in Pediatrics at UT Southwestern and Children’s Medical Center in Dallas, who pulled no punches in his assessment of the new study.

“The basic issue is intriguing, but bias, methodological flaws, conflation of statistical with clinical significance, and a population that is a far cry from the general population of U.S. children in Head Start render this study essentially trivial,” Dr. Flores said.

(To their credit, the researchers do acknowledge the homogeneity of their sample and report that they are currently evaluating a modified version of the program for Spanish-speaking families.)

While Flores went on to identify a long list of what he considered to be the study’s many methodological weaknesses (including the use of a p-value of 0.1 rather than 0.05 as a cutoff for comparisons between groups), he ended by questioning the practical import of the research:

Most importantly, statistical significance, rather than clinical significance, was the focus. This is crucial because statistically significance can be found for very small, clinically insignificant differences. Indeed, the incremental improvements for the “Parents and Children Making Connections – Highlighting Attention” group were mostly minute, and in several cases, the post-test scores of the PCMC-A group were about the same as the “Head Start” and “Attention Boost for Children” groups.

The idea here is that even if certain comparisons between a treatment and its controls are deemed statistically significant, the differences may not be big enough to warrant practical changes, policy shifts, or increased spending on a program.

Dr. Flores’ critique suggests a couple of points. The first is more of a reminder: For journalists who find themselves sifting through the academic literature, getting independent takes from experts in the field is essential in interpreting the science. You don’t necessarily have to adjudicate differences between a study’s authors and its critics, but you’d be remiss for not at least including the latter’s criticisms.

Second, the particular type of intervention outlined in the Oregon study may yet prove rather helpful – building the skills and abilities of parents is a hot topic these days – but that conclusion awaits more rigorous research.

“What is needed is a rigorous, multi-center randomized, controlled trial over the course of year focusing on parenting skills, home routines, and children’s skills,” said Dr. Flores. “This would have the potential to be an important contribution.”

Image by Lupuca via Flickr