Retraction, the process by which a journal withdraws its research after publication, is an essential tool for eliminating erroneous or fraudulent studies from the scientific literature. But a preprint published on medRxiv June 30 shows that retraction may not work as intended: Retracted articles on clinical research on COVID-19 have been cited more than 1,000 times, largely uncritically, indicating that conclusions drawn from Unreliable research may continue to affect the literature and scientists’ understanding of the disease.

In addition, many of these citing articles were submitted for publication after the original articles had been withdrawn, raising concerns about author and journal citation standards. Research published in PNAS June 14 also found that most of the articles that are subsequently retracted had already been widely circulated on news sites and social media before being taken off the record, further fueling concerns that the tool is ill-equipped to limit the spread of misinformation.

The scientist spoke with preprint co-author Gideon Meyerowitz-Katz, an epidemiologist at the University of Wollongong in Australia, about the implications of retracted article citations and potential solutions, as well as broader production issues and publication of research.

The scientist: In this preprint, you reported that retracted articles on COVID-19 are still cited. How often does this happen? Were you surprised by the results?

Gideon Meyerowitz-Katz: What we found was that the modal count – the most common count – of citations that a retracted article got was zero. But the median was between four and seven, depending on how you split it. Most retracted articles were cited at least once. The citation of retracted articles is very common and rarely critical.

While scientists are truth seekers to some extent, we are also human and can fall prey to confirmation bias.

This [preprint] was specific to COVID, so it’s hard to draw conclusions about all other types of retracted articles. It is plausible that there is a difference and that it is specific to the pandemic we find ourselves in. But I would say in my experience it seems pretty common for non-COVID research as well, so I wasn’t surprised. I think the quote tends to be cavalier in a way. And I think it’s extremely rare for people to look at their citations and check if there are any issues with the articles or if they’ve been retracted before submitting them.

TS: In the preprint, you mentioned that even before these studies were retracted, some of them are obviously of poor quality or even include impossible data values. Why do you think they are always quoted?

GM-K: I may have a slightly cynical view that people don’t pay much attention to citations in scientific literature. . . and it’s very rare that people check your list of references carefully during the peer review process. And I don’t think people notice if the articles are bad, as long as they agree with their opinions. While scientists are truth seekers to some degree, we are also human and can fall prey to confirmation bias.

See “When researchers sound the alarm on problematic articles”

For me, the biggest problem isn’t people citing retracted research – because I can pretty much see how you read a research paper and put it in your citation manager or you download the PDF and you don’t. ‘ll ever check if it has been retracted in the future. I mean, why would you? You just assume it won’t be retracted. However, the fact that people are citing research that’s so bad that it’s obviously going to be taken down pretty soon—and people routinely do that, with no quality control in that process—it’s pretty concerning, especially because the majority of articles that should be retracted probably aren’t.

See “Eye for Manipulation: A Profile of Elisabeth Bik”

TS: What are some of the potential implications of this, particularly during COVID, in terms of scientific knowledge and clinical practice?

GM-K: There has certainly been a real impact of all of this during COVID. . . . Some of these articles were clinical research cited in systematic reviews or meta-analyses that informed clinical practice. There were a number of repurposed drug articles that were taken down, such as favipiravir, hydroxychloroquine and ivermectin, and those certainly got people prescribing drugs.

Even if we completely fix the citation of retracted articles, it will not solve the underlying problem, which is that there are a lot of very poor quality articles.

See: “No reduction in COVID-19 hospitalizations, deaths with ivermectin”

TS: What strategies could researchers or journals use to ensure that these retracted articles are not cited? And what are the potential barriers to implementing these strategies?

GM-K: In theory, if researchers carefully check their references before submitting [the paper], that would not happen; you could prevent it very easily. But in practice, it’s a systemic problem, and I think systemic solutions are needed. [One solution] which is in fact already offered by some journals is an automatic note. Every time someone submits [a] manuscript. . . this automatic mechanism simply sends an e-mail saying: “You cited retracted articles, are you sure you want to cite these articles?” The process of scientific publication already requires a lot of effort on the part of scientists…[unpaid] work with little or no benefits. So having an automated tool that doesn’t require [more work] is vital if we are to prevent this from happening.

TS: There is also the perhaps trickier issue of those shoddy but not necessarily retracted articles that are quoted. How can we fix this problem?

GM-K: I think it’s a much trickier issue because citing retracted research, ultimately, demonstrates that there’s a lot of shoddy work in the scientific literature now. But even if we completely fix the citation of retracted articles, it won’t solve the underlying problem, which is that there are a lot of very poor quality articles. There are papers that are still published that are obviously false, that obviously never happened, or at least where the datasets contain mathematical impossibilities. . . whose editors have stopped responding to emails and which will probably never be retracted. It’s not even that rare.

See “Document proposing COVID-19, magnetism link to be removed”

Our article sheds light on a specific question, but the larger issues with the scientific literature, I think, are much more difficult to resolve.

TS: Are there ways to raise widespread awareness of this without further undermining trust in science?

GM-K: I think there’s a pretty good awareness of this because it’s a meta-research question that’s been studied for a long time. It’s just that journals don’t have the responsibility to ensure that the content they publish is true and real. They have a responsibility to ensure that they are peer reviewed. But it can mean many different things. And for some journals, that’s not a very high bar to reach. Peer review rarely verifies manufacturing. . . . Some of the biggest journals like Science and Nature employ statistical examiners [who] are supposed to check the numbers to make sure they make sense. But if you look at the average newspaper, it’s unlikely that someone would add up the columns of a table to make sure they add up, which I’ve found many articles don’t do correctly.

This relates to the issue raised in our article that there is no individual or group whose responsibility it is to ensure that retracted research is not cited. The editors are there to ensure that the research is peer-reviewed and interesting enough to be published in the journal, but peer review is by no means designed to detect such issues.

Editor’s Note: This interview has been edited for brevity.