Reporting on Evidence Requires More Than Spin
Reporting on evidence based medicine is tricky. Although randomized controlled trials are sometimes considered the final word on EBM, the reality is they may not be. A study published in PLoS in December 2012 found that randomized controlled trials are sometimes misrepresented in press releases and subsequent news coverage.
The Science Media Centre UK recently promoted one of a continuing series of papers based on a large RCT, however the results remain controversial among researchers - as does the reporting.
Although the SMC UK did offer a number of experts as sources, they apparently chose not to include any outside experts who questioned the results and failed to mention industry ties for one of the sources they listed - as well as the authors. Industry ties, whether they be Big Pharma or disability insurers, are not an automatic conflict of interest, but transparency is crucial as is balance in reporting.
Variables count
Whether the results of a trial can be extrapolated to everyone depends on how the many variables were handled. Very few trials cover all variables. For example if an RTC studies white, middle-aged, males does that mean the results can be extrapolated to women, children or different races? It is far too easy to oversell a treatment, particularly in the headline, when the journalist fails to ask hard questions or use independent sources.
In the case of this trial, how the cohort was defined could presumably impact how results were interpreted and who they applied to. When the initial study first came out U.S. journalist David Tuller MPH, wrote the standard article for the New York Times. However, after an outside expert unaffiliated with the trial contacted him about some of the issues, Mr. Tuller wrote a second piece about a key problem with the trial conclusions. The Wall Street Journal coverage followed a similar path.
Why was this potentially important? A number of large patient surveys showed that the treatments touted may be harmful for some patients. Although the study researchers insist they are not, the potential failure to include the patients most likely to be harmed in the study means there is not yet an evidence base for the safety of the treatment in such populations. While journalists themselves don’t determine if a treatment is safe, safety concerns should always be taken seriously and once again outside experts with concerns should be included. And as another U.S. researcher said, the benefits were modest at best even in cohort studied in tertiary setting where the patient had to be well enough to repeatedly travel to recieve treatment. This also limits generalization as patients too ill to travel were not studied.
So was it efficacious or effective?
Efficacy is when treatments are compared in the same study. In this case, there were also some questions raised regarding whether appropriate comparisons were made and whether the application of treatments matched what other researchers use and study. Once again, interviewing outside researchers who study the treatment these authors compared theirs to would have clarified issues regarding suitability and efficacy. Whether these treatments are effective in a standard clinical setting is not known.
How were results measured?
Most clinical trials publish their protocols first followed by the trial at its conclusion. This allows journalists who read both to determine whether endpoints were changed and ask why, if they were. As well, not all endpoints are meaningful or may use a different standard than similar trials.
Regarding the original trial the authors wrote a letter to the editor protesting the second article Mr. Tuller wrote. Mr. Tuller's reply to the letter clarified why the trial as it was done did not meet the gold standard of evidence-based medicine. Mr. Tuller could not have done that had he not read the full paper and protocols as well as interviewing appropriate outside sources.
In this case, perhaps even more concerning than the dearth of appropriate coverage was that the SMC UK was involved in the spin.
Spin can come from many different sources and may be best countered by reading both the protocol and the full paper, using independent expert,s including researchers in the field whose research results are different than that of the authors and providing meaningful context, as Mr. Tuller did.
Image by Jaqian via Flickr