By Tara Haelle
Deciding whether to report on a clinical trial or other medical study requires considering factors that range from the study’s news value to the strengths and weaknesses of the study itself. The former is far easier for journalists to determine than the latter.
This tip sheet is the first in a series pointing out red flags that journalists should watch for in the studies they cover. Seeing one in a study doesn’t mean it shouldn’t be covered. In fact, sometimes a study with a lot of red flags is worth covering if it has strong news value but runs the risk of being misinterpreted in other articles. In that case, covering it may present an opportunity to ensure an accurate representation of the study, its limitations and implications compared with other coverage that may or may not cover it so judiciously.
Small Population (small n)
Generally speaking, the smaller a study’s population, the lower the statistical power. It can be harder to determine whether the finding is real or a result of chance. This is a broad generalization, since other factors play a role in how reliable findings might be in a small study. And what is considered “small” will depend on the type of study. A larger population is needed for epidemiological and observational studies because more confounders are likely to interfere with the factors under study.
In a randomized controlled trial (RCT), however, much smaller numbers can be used because the experiment is controlled. A phase one clinical trial evaluates a drug’s basic efficacy and general safety. It almost always will involve smaller patient population than a phase three trial, in whch researchers are evaluating a wider range of safety concerns, as well as testing the efficacy of doses based on what was seen in the phase one and phase two studies.
How small is too small a sample? Anything less than 100 participants in a RCT should raise eyebrows, although you may only see 20 to 50 participants in studies involving a complex intervention, such as a sleep or nutrition study in a controlled environment. Before covering a study with an “n” of less than 20, you’d want the experiment to be particularly complex or novel, or to be exploratory or a proof-of-concept study. For retrospective and prospective epidemiological cohort studies, several hundred or several thousand participants are ideal.
For case-control studies, at least 50 to 100 cases at a minimum is preferable, although it depends on the case under study and how common it is. Studies of a rare disease, for example, will necessarily have a low number of cases, whereas case-control studies for cardiac conditions should have much larger populations.
Significance
A study’s findings are weakened when they lack statistically significance or clinically significance. Studies often will report data that is not statistically significant, but the researchers may attempt to play them up by noting they “trend” in a particular direction. “Trending” in this context is essentially “almost but not quite” and probably not worth reporting on except to point out that the findings were not significant.
Even when the findings are statistically significant, be sure they are clinically significant, or if not, that there is another good reason to report about them. Effect size can be one indicator of clinical relevance. A statistically significant change that is so small as to be clinically irrelevant probably isn’t that newsworthy.
If some findings are significant and others are not, look at whether the significant ones relate to the primary endpoint, which is the most important question being asked in the trial. Some studies will play up clinical significance in a secondary endpoint, subgroup analysis or incidental finding when the primary endpoint did not yield a statistically significant finding.
Journalist Emily Mullin points to a study by Singapore-based TauRx of the Alzheimer’s drug LMTX as a good example of a study that missed primary endpoints but still was promoted as “promising” in press releases. Check out Matthew Herper’s coverage in Forbes about that study’s weaknesses.
Confounding
Beware of studies where there is too much room for confounding. Data for observational studies, for example, may be reported daily but there’s still a risk that the findings can be confounded by other factors. “I always take a minute to brainstorm alternate explanations for what they found, and then check whether they tried to find ways to control for those,” journalist Beth Skwarecki said.
This risk especially true for studies attempting to link some early life factor to a health outcome many years later – especially if few or no other studies have reported the same findings and the biological mechanism for a link is weak or nonexistent. Other observational studies to consider skipping include:
- Those that do not control for confounding variables at all.
- Those that control for too few confounding variables when others may be particularly notable.
- Those that look at many outcomes or poor methods for identifying an exposure or an outcome, such as using only a zip code to estimate air pollution exposure. Studies on pregnancy, infancy and early childhood, mental health conditions and rare diseases can be the worst offenders of this study flaw.
Methodology
Be skeptical if a study’s methods section lacks sufficient detail. Some studies require analyzing copious amounts of data or multiple variables or outcomes. The complexity of the analysis should be reflected in the methods section. Ask the authors about their methods or run it by an outside source. Or do both. For a best-case scenario, check with a biostatistician to ensure that the particular statistical methods used were appropriate for the analysis they were performing.
Tara Haelle, AHCJ’s topic leader on medical studies, writes blog posts, edits tip sheets and articles and gathers resources to help our members cover medical research. If you have questions or suggestions for future resources on the topic, please send them to tara@healthjournalism.org.
