How to think about impact factors: A researcher’s perspective

Back in February, my tip sheet on the AHCJ site about evaluating the quality of a study mentioned that a journal’s impact factor may be considered in assessing whether to report on a study. While mentioned as one factor among many — and never a definitive one — it inspired a fair amount of debate on Twitter about the value, or lack thereof, of impact factors, especially given the rise of new open-access journals that haven’t been around long enough to have a high impact factor.

To add to the discussion, I reached out to two individuals with different perspectives— largely related to their differences in goals, values and needs in the spheres of research and journalism. Today’s post features a Q&A with Hilda Bastian, Ph.D., scientist, blogger at Absolutely Maybe on the Public Library of Science (PLOS) site and cartoonist at the wonderful and educational Statistically Funny site. Bastian also is a chief editor at PubMed Health and PubMed Commons, but her words here represent only her own views and not the views of any organization, including the National Institutes of Health. Bastian comes from a background of scientific research.

Hilda Bastian

Hilda Bastian

What are the limitations or problematic aspects of using impact factors to judge the quality of a journal?
The impact factor of a journal measures a type of prestige and age, not quality. And it’s not prestigious to everybody. In fact, it’s the opposite to many excellent scientists, who boycott non-open access “glam journals” on principle. The impact factor is open to gaming, too.

Being in a high impact factor journal is not even close to a guarantee of high quality, any more than not being in one is a guarantee of lower quality. And that’s the problem. The set of journals with the highest impact factor does not equal the best journals, even though some of them are among the best. Being in a high impact factor isn’t even a reliable indicator that the specific article has a higher chance of being highly cited.

The risk of thinking that the high impact factor of the journal means quality of everything inside it is that you give an article a free pass or extra brownie points it doesn’t deserve. If you reject the importance of an article just because the journal has a low impact factor, you could be missing an incredibly important article — especially if the journal is a young one.

For example, you would have rejected one of the most important and high profile papers in science in the last 20 years: John Ioannidis’ article on why most research findings are false. And if you cover articles only because they are in high impact factor journals, you could have covered some of the highest profile cases of fraud and misconduct.

If a journalist is looking at a journal’s impact factor while covering a study, what important caveats should they keep in mind?
I don’t think a journalist should look at a journal’s impact factor while covering a study. I think it should be avoided. If a journalist is going to do it, and they give the advantage of more press coverage to an article because of the impact factor of the journal, they have fallen for a kind of PR. And they have become part of one of science’s most serious problems. Women, scientists of color and non-Americans, and early career researchers are all systematically under-represented in those journals — as are many scientists who are trying to reform this situation — if it’s also not an open-access journal.

Impact factor journals are one of the key influences diverting journalists’ attention away from important research because of the advantage of prestige and resources. That’s not in the public interest. I hope journalists can act as a force against prestige-bias in society, not be a key tool contributing to it.

That doesn’t mean that I think journalists should not cover articles in high impact factor journals. While boycotting them is an important choice for many scientists, it would be disastrous if journalists did it. It’s a completely different situation.

What advantages exist in looking at impact factors, if any?
I really thought about this. The only one I can think of for a journalist is that it puts you on notice that anyone you’re asking about the quality of the article, might be giving it a bit of a free pass because of the name of the journal.

Leave a Reply