Covering a controversial study: How to dig deep on a deadline

From the moment I saw the study — and editorial and editor’s note — among JAMA’s embargoed studies, I knew it would be a doozy. Certain topics arouse controversy simply by their existence, and water fluoridation is very high on that list.

So when I was assigned to write about the JAMA Pediatrics study (Reminder: AHCJ members get free access to the JAMA Network.)  finding a link between prenatal fluoride exposure and reduced IQ in preschoolers, two things went through my mind: One, this is going to be covered horribly by some outlets and likely create unnecessary anxiety among parents, especially pregnant women (who have enough to worry about when it comes to do’s and don’ts). Two, I need to be one of those who gets it right.

The first thought wasn’t a lack of faith in my health reporting colleagues, who I expected would write about the research with due diligence. Rather, it was recognizing that studies like this one easily get misrepresented or blown out of proportion when covered by news outlets who assign the stories to general reporters or those otherwise not experienced in covering medical research. Indeed, the quality of coverage varies greatly (and is sometimes diminished primarily by an unfortunate headline):

• Is fluoride in water toxic to babies’ brains? A rigorous study raises alarms (The Philadelphia Inquirer)
• Study raises questions about fluoride and children’s IQ (The Washington Post)
• Study prompts call for lower fluoride consumption by pregnant women (Reuters)
• Drinking fluoridated water during pregnancy may lower IQ in sons, controversial study says (Science)
• Study of fluoride during pregnancy and children’s IQ raises questions but draws criticism (ABC)
• Can Maternal Fluoride Consumption During Pregnancy Lower Children’s Intelligence? (NPR)
• New study raises questions about how fluoride affects children’s development (CNN)
• Is Fluoride in Drinking Water Safe? A New Study Reignites a Long-Standing Debate (Time)

These are just a selection of the headlines that I chose at random. I’m not going to call any out because I don’t always think that’s helpful, and several of these did an excellent job of covering the research responsibly within the stories. Instead, I thought I’d share how I approached, reported and completed my own article at Medscape, “Prenatal Fluoride Exposure and IQ in Kids: Is There a Link?,” in case seeing my process is helpful for others when there’s another potentially controversial study to cover.

A couple of caveats: I had the luxury of an editor who allowed me to write a very long story (about 2,500 words), and I was already very familiar with fluoride research because I reviewed it for a section in the book I coauthored with Emily Willingham, “The Informed Parent.” That said, I got started on the story after 5 p.m. ET on Thursday, and the embargo lifted at 12 p.m. ET on Monday.

1. I skimmed the study, skimmed the editorial and read the editor’s note. I did not read the study in detail at this time.
2. I opened all the recent fluoride studies referenced in the editorial to get a sense of recent research (all published since I’d last looked at the evidence). I used the references in the editorial instead of the study because it was more likely to note only the most recent and relevant studies. I wouldn’t have time to look at everything.
3. I instantly saw that a 2017 Mexico City study on fluoride and IQ shared an author with the new study, and two previous studies linking fluoride and ADHD in children also had overlapping authors with the new study. One of these ADHD studies, also in Mexico City, involved the new study’s lead author, corresponding author and a third author. Seeing these overlaps is helpful because it brings up the possibility of ideological bias: Do these authors frequently publish findings linking neurodevelopment or cognitive problems with fluoride exposure? Are there other similar papers by different authors showing similar findings, or are they the only group out there finding these results?
4. I emailed the corresponding author, Christine Till, of the new study for an interview.
5. I also quickly saw that another related study from New Zealand had negative findings (that is, they found no correlation between fluoride exposure and reduced IQ), but I noted that the one I was covering, like the Mexico City one, used a biomarker (urinary concentration) and the New Zealand one did not. Biomarkers aren’t perfect but are more reliable than pure geographical correlation, as is used in most fluoride studies.
6. I emailed the corresponding author for the New Zealand study. Knowing that time zones might become problematic for an interview, I also looked up and emailed the only U.S.-based author on the paper.
7. I felt it important to find outside experts likely to be skeptical of the study. I looked up the Medscape article about the Mexico City study and found Scott Tomar, a public health dentist and oral epidemiologist at the University of Florida. I emailed him requesting an interview, though I noted he consulted for the American Dental Association.
8. I looked up other past news stories about that study to see what other outside experts had been used. Honestly, this was kind of depressing because few of the stories — at least those coming up on Google News — included outside experts, and those that did had experts that didn’t offer much insight into the study.
9. I read the study more closely, this time formulating questions for my interviews. (See bottom of post.)
10. I searched the HealthNewsReview.org website to see if they’d written about coverage of fluoride before. Indeed they had, so I read this excellent piece by Kevin Lomangino on “Four questions to ask when reading or writing about water fluoridation and IQ.” Nothing there was new to me, but it’s always helpful to review best practices. I also knew I’d be asking a LOT of questions about the methodology of this study, which would be key to assessing the reliability of its findings.
11. I went to sleep Thursday night.
12. I interviewed Tomar, the University of Florida dentist, on Friday morning and asked him for other folks to contact.
13. He recommended Eugenio Beltran, an adjunct professor of epidemiology and health promotion at New York University, and Steven Levy, a professor of research, department of preventive & community dentistry at the University of Iowa, both of whom I immediately emailed. I was conscious that it’s August — the hardest time of the year to reach researchers — and I didn’t know who would respond, especially given my deadline (Sunday afternoon).
14. I looked up Levy’s past publications to see if there were any coauthors who might be worth interviewing. In the process of doing so, I noticed he’d worked with one of the authors on the study I was covering.
15. A coauthor on a 2011 study of his was a military public health dentist in Texas. I emailed him because a) military folks tend to be conscientious about regularly checking email, b) he specialized in public health dentistry, where fluoride is positively regarded, and c) he was in Texas, and I’m hyperconscious of time zones when it’s a Friday and my deadline is over the weekend. The West Coast is my friend.
16. AHCJ President Ivan Oransky, M.D. — also vice president of editorial at Medscape — always recommends having a “biostatistician in your back pocket” for questions about studies. I’m not as good about this as I’d like — I need to develop more professional relationships with biostats folks — but I did know two people right off who might be helpful because both had co-presented on medical studies with me at AHCJ conferences: Perry Wilson and Regina Nuzzo. I emailed both for help in looking at the biostats in the paper and included this in my email: “My main concern is this: even given the usual limitations of observational studies like this one, are the methods appropriate enough, the data strong enough, the effect big enough, the stats correctly done enough to support their conclusion?”
17. Biostatistician Wilson was actually writing a commentary about the study already but provided an excellent summarizing quote about the study that I included. Nuzzo brought up a whole host of considerations that I’d never have thought of and which were immensely helpful in my interviews and follow-up questions with the author.
18. During our interview Saturday midday, corresponding author Till mentioned the JAMA podcast about the study that I’d missed. I almost never look at press releases and, in this case, it meant I didn’t see the podcast among the JAMA embargoed materials. I listened to the podcast after our interview and got some helpful quotes from it from two JAMA editors.
19. Till, the corresponding author, also sent me a list of five researchers I could contact who said they’d be available to comment on the study. Usually, I’d be extra cautious of outside experts recommended by an author, but with this study, I didn’t think it was possible to talk to too many people in the time I had, and I wanted to be sure I wasn’t succumbing to my own confirmation bias by only seeking out critics. And, again, it’s harder to find folks in August, so I thought it was worth contacting some of them.
    I reached out to three of those Till recommended, plus the study’s lead author. By now, it was already Saturday afternoon, so I included five questions in my email in case they weren’t available for a phone interview but might be able to email some comments.
20. The New Zealand researcher, Jonathan M. Broadbent, responded and was available for me to call. Though most of our call was off the record, it was helpful to speak with him.
21. One of those that corresponding author Till had recommended, Howard Hu, who also led the Mexico City study, called me (after descending a mountain with his sons while they were on vacation!) and was very helpful in providing further comments about the study.
22. I received a response from another researcher who preferred not to be quoted but who answered several of my questions in email that helped me in thinking about the study and framing the story.
23. Saturday night, I began writing the story. In the midst of writing it, I received a response from corresponding author Till to some follow-up questions I had as a result of Nuzzo’s comments. When Till responded, she cc’d the lead author, Rivka Green, who also responded. Though I didn’t quote Green in the story, the follow-up answers from her and Till were helpful.
24. While writing the story, I realized Tomar had commented on fluoridation policy but not on the authors’ recommendation that pregnant women reduce their fluoride intake, so I emailed him to ask if he had any thoughts about that recommendation. I also asked for any additional disclosures he had aside from consulting for the ADA and FDA.
25. While writing the story, I made sure to include the top-line findings and the authors’ recommendations, but I also emphasized the methods (both strengths and weaknesses) and important questions about the study’s findings and implications — including possible reaction from the public. I included much more detail about the methods than I normally would when covering a study like this, but I felt it was important to give the extra detail and scrutiny to the methods since they were so important to interpreting an observational study like this that cannot show causation but which has the risk of causing anxiety in the public.
26. I filed the story very, very late Saturday night (really, early Sunday morning).

Final thoughts: I devoted much more time to this story than I typically would spend on a single-study story, and I talked to more than twice as many experts as I normally would have. But it was a study about a very controversial issue that was complex, greatly affects public health and had the potential to arouse anxiety and/or inspire sharing misinformation. I was fortunate to have already been familiar with fluoride research, which helped in writing the story on a very tight deadline.

I’m grateful I was given the space to write so much about the research and include so many experts’ comments. Even if writing a much shorter story — a more typical consumer piece of 800-1,000 words, for example — I would still have contacted as many people. I’d have left out most of the details about the methods (these are more relevant to the trade audience I was writing for but are less relevant to consumers) and would have greatly shortened the sections on public reception.

It would be awesome if I could devote this much time and thought to every study I cover. That’s not possible, but when a study merits it, it’s worth investing the time and energy.

The initial questions I drew up for the author interview with Till (though I asked additional follow-up questions during the interview):

1. To what extent were these findings surprising, particularly after having conducted the Mexico City study? Why or why they not surprising?
2. Which unmeasured confounders, or not-measured-as-well-as-you’d-have-preferred confounders should be noted that may affect the analysis and/or basic findings?
3. Would it have been possible to measure blood concentration instead of urine concentration of fluoride? Why was urine chosen given potential limitations, such as only being a proxy for fetal exposure/crossing the placenta and possible contributors to Ph differences that might (?) affect the findings?
4. What concerns, if any, do you have about how this study’s findings might be interpreted by physicians and/or the public at large?
5. What actions, if any, do you hope might be taken following publication of this study, other than other researchers conducting similar studies for replication?
6. Anything you want to add?

These were the follow-up questions I sent Till after our interview after receiving Nuzzo’s comments:

1. Would it be possible in a future study, and would it be more potentially accurate/precise, to measure blood instead of urine concentration since the latter is less susceptible to Ph changes and would better estimate true prenatal exposure/placenta crossing? (I’m assuming there was a reason you were unable to use blood levels in this study and/or chose urine instead of blood?)
2. You’ve somewhat addressed this already in simply noting caution about interpretation and discussing population-level effects, but for the sake of greater precision: Given the natural variation and margin of error that occurs with IQ measurement (particularly in youth), how practically significant is the IQ difference in the findings when the majority of mothers are taking in 1mg or less daily and the difference in fluoridated vs non-fluoridated areas is 0.29 mg/L (correct?)? It seems moving from a non-fluoridated area to a fluoridated area would only drop IQ on average by 1.3 points (4.49 decline x 0.29 difference in regions = 1.3). (Please let me know if any of the math is wrong there!)
3. We already discussed the greater potential susceptibility for prenatal toxicant exposure in boys. Any idea why there appears to be a non-significant trend toward a protective effect in the girls? (as opposed to a fully flat effect)
4. Finally, in reference to your recommendation that pregnant women be conscientious about their fluoride intake: Considering the estimated drop of 3.66 points for each additional 1 mg increment of daily fluoride intake, that suggests that drinking, say, twice as much tea than others on average most days might drop a child’s IQ by 3-4 points. That seems … really extraordinary. Is that indeed what the findings imply?

These are the questions I included in my emails to outside experts requesting a phone interview (in case they weren’t available on the phone):

1. Please let me know any disclosures you have and how you would prefer your title and credentials to appear.
2. Can you speak to the quality and rigor of the methodology? What are the biggest limitations that might give pause in considering the findings? To what extent using urine instead of blood concentration limit the reliability of the findings?
3. Looking at their raw data (scatterplot, etc.), do you feel the data supports their conclusions? Why/why not?
4. Do you think they controlled adequately for the appropriate confounding factors? Are there any potentially important ones they missed? (Or any other comments about their covariates?)
5. Are there any clinical implications of this study and/or the research to date at this point in time? What would you say to providers and/or parents on how to think about this study?
6. The heart of my questions comes down to this: even given the usual limitations of observational studies like this one, are the methods solidly chosen and executed enough, the data strong enough, the effect big enough, and the stats adequately analyzed enough to support their conclusion? And if so, what does that mean in the bigger picture?

Related

• Fluoridation findings prompt measured response from professional organizations

Correction: A previous version of this post erroneously reported the number of shared authors between the current and previous fluoride studies.