By Tara Haelle
In a recent 13-part series about databases frequently used in medical research, JAMA Surgery provided a 10-item checklist for researchers to use to ensure they are using the most appropriate database in the most appropriate way for their research.
The article, “A Checklist to Elevate the Science of Surgical Database Research,” is intended for researchers. However, it doubles nicely as a list for journalists to use in thinking about questions they should ask researchers who use large databases in their research.
Below is each item in the JAMA Surgery list in italics followed by what journalists might think about when reading a study using a large data set. Keep in mind: it is certainly not possible to ask all these questions of every study you report on, especially on a tight deadline. But if you have more time with a study, or if it’s a particularly controversial or “big news” item that deserves extra scrutiny, it may be worth considering more of the items below. Or, if you are digging into many different studies for a book or investigative project, the considerations below may be helpful guidance.
1. Have a solid research question and clear hypothesis. Consider using the FINER (Feasible, Interesting, Novel, Ethical, Relevant) or PICO (Patient, Population, or Problem; Intervention, Prognostic Factor, or Exposure; Comparison or Intervention; Outcome) criteria to develop these.
JOURNALISTIC CONSIDERATIONS: Is the question the researchers are asking clear and specific enough? Is their hypothesis stated clearly and precisely, or is it vague? Is the question or hypothesis they present in the paper the one they started with? (You can ask during the interview. If either has changed, why?)
2. Ensure compliance with the institutional review board and data use agreements.
JOURNALISTIC CONSIDERATIONS: Did the researchers make sure they explained in their methods section whether they needed IRB approval (often it’s unnecessary with large datasets) and if any restrictions existed on using the dataset?
3. Conduct a thorough literature review. Use a reference management program for ease in manuscript development.
JOURNALISTIC CONSIDERATIONS: Did the researchers provide adequate background information on why they did the study? Does it seem thorough or cherry-picked? (Is everything pointing in one direction, or do they acknowledge contradictions, questions and nuances in the existing evidence base? Most questions involve nuance.) Are there any references cited that seem off or make a claim that seems surprising or dubious? If so, you might want to check that citation.
4. Make sure this is the best dataset available and that it has the appropriate variables to answer your research question.
JOURNALISTIC CONSIDERATIONS: Why did the researchers choose this dataset for their study? Do they provide an adequate rationale? If not, ask them why they chose it (and maybe why they didn’t mention their reasons in the paper). Are there limitations to this dataset that make it particularly problematic for the question they’re asking?
5. Clearly define the inclusion criteria, exclusion criteria, and outcome variables. Use a flow diagram to describe final patient selection.
JOURNALISTIC CONSIDERATIONS: Did they include the inclusion and exclusion criteria, the outcome variable and the flow diagram in their methods section? Do the criteria seem appropriate? Any criteria items seem odd or extraneous? Scrutinize the flow diagram and be sure the numbers add up. Look for high drop out rates or excluded people.
6. Identify potential confounders and use risk adjustment to minimize bias. Consider using a directed acyclic graph to represent potential associations. Avoid use of causal language in reporting results of these observational studies.
JOURNALISTIC CONSIDERATIONS: Did they avoid implying causation in describing any associations? What kinds of bias might the study be subject to? (They should address at least some possible bias in the limitations section and possibly in the methods section.) Did they make adjustments to reduce bias or account for confounders? Are there any confounders (particularly confounding by indication) they should have considered but didn’t? All of these could lead to possible questions to ask the researchers or outside sources.
7. Ensure that the data variables have not changed over time. If so, account for this.
JOURNALISTIC CONSIDERATIONS: In long-term studies, things such as a participant’s geography, occupation, education, diet, exercise levels, comorbidities, socioeconomic status or other characteristics could change. If it’s long-term enough for those possibilities, did they consider that and adjust for it if necessary?
8. Ensure that competing risks are identified and addressed.
JOURNALISTIC CONSIDERATIONS: Competing risks occur when one outcome precludes another. For example, if looking at re-hospitalization rates following a particular condition or event, the researchers need to consider whether a person was not re-hospitalized because they died rather than fully recovering. See if they address these kinds of possibilities in their methods, discussion or limitations sections.
9. Ensure that data issues, such as missing data, are discussed and that any sensitivity analyses or imputations performed are reported in a clear and cohesive way.
JOURNALISTIC CONSIDERATIONS: With such large datasets, there could be large subsets of records missing key data, such as demographic information, admission or discharge dates, comorbidities, disease characteristics, etc. The researchers should have addressed whether there was missing data among their dataset and how they dealt with it: did they avoid using that variable altogether? Did they exclude all those records? If they excluded them, could that introduce bias if all the records with missing data shared an underlying factor that differs from records with complete data? Did they make statistical adjustments to account for missing data or excluded records?
10. Ensure that your article has a clear take-home message that addresses how your research advances current knowledge and has important policy or clinical implications.
JOURNALISTIC CONSIDERATIONS: This is one of the most helpful questions to ask the researcher: what is the most important take-home message from your study? You can also ask how their findings are clinically significant or whether they have the potential to change clinical practice, some type of government policy or individual behavior. They should include a take-home message in their study, but asking about it allows them to elaborate and is often a good starting place in an interview.
