High-profile coronavirus retractions raise concerns about data oversight

Two weeks after a high-profile paper in The Lancet¹ reported that the antimalarial drug hydroxychloroquine might be dangerous to patients with COVID-19, three of its four authors have retracted the work because they were unable to independently verify their data set, a large proprietary collection of electronic health records analysed by Surgisphere, a US company.

On the same day, 4 June, the researchers and other co-authors retracted a paper in the New England Journal of Medicine (NEJM)², for the same reason. That study, published a month ago, had looked at the impact of certain heart medications on people with COVID-19, and found no safety concerns.

Scientists say the affair raises serious questions about the way researchers and journals evaluate the data underlying papers that they publish, and may complicate the effort to trial drugs during the coronavirus pandemic.

“This whole event is catastrophic — it is problematic for the journals involved, it is problematic for the integrity of science, it is problematic for medicine, and it is problematic for the notion of clinical trials and evidence generation,” says Ian Kerridge, a bioethicist at the University of Sydney, Australia.

Both papers relied on proprietary data analysed from electronic health records that were apparently gathered from hundreds of hospitals around the world by Surgisphere. But after critics raised questions about the studies, the firm did not make its raw data available to third-party auditors for validation. According to the retraction notice in the Lancet, Surgisphere was concerned that transferring the data would violate "client agreements and confidentiality requirements".

“Since we do not have the ability to verify the primary data or primary data source, I no longer have confidence in the origination and veracity of the data, nor the findings they have led to,” said Mandeep Mehra, a cardiologist at Brigham and Women’s Hospital in Boston, Massachusetts, who was the lead author on both studies.

Sapan Desai, the founder of Surgisphere, in Chicago, Illinois, and a co-author of the studies, declined to comment to Nature on the retractions and on concerns about the quality of the data behind the studies. Desai signed the NEJM retraction but not the Lancet one.

After the Lancet study appeared, some regulators had paused enrolment in clinical trials of hydroxychloroquine as a treatment for COVID-19. The drug, which is cheap and easy to administer, has been widely endorsed in the pandemic, including by US President Donald Trump, despite scant evidence of its effectiveness. Some of the halted trials, including one led by the World Health Organization, are now starting again. But researchers say they’ve lost time and that enthusiasm to take part in the trials may be waning.

A third study authored by Desai, Mehra and others that used Surgisphere data, which claimed to find a large reduction in COVID-19 mortality when patients were given the anti-parasitic drug ivermectin, has disappeared from the social-sciences preprint server SSRN, where it was first posted on 6 April; a second version was posted on 19 April. Mehra told Nature that he removed the study because he "did not feel it was ready for peer review".

Although the paper was not published in a peer-reviewed journal, it has nonetheless contributed to a surge in the popularity of ivermectin in South America, says Carlos Chaccour of the Barcelona Institute for Global Health in Spain, who is running a clinical trial of ivermectin and had raised questions about the preprint’s data.

Confidential data

Bioethicists say the retractions, and the pulled preprint, raise questions not only about the quality and nature of Surgisphere's data, but also about why the paper’s other authors agreed to work with a large data set they couldn’t validate, and how the work passed peer review at prestigious medical journals.

The hydroxychloroquine study, published on 22 May, had purportedly analysed electronic health records gathered on 96,000 patients in 671 hospitals across six continents. But critics soon raised questions about oddities in the data, and asked for more details on its origins. By 28 May, 120 researchers had signed a letter to The Lancet highlighting their concerns. The questions also cast doubt on the NEJM study, which relied on Surgisphere data that apparently covered almost 9,000 patients across 169 hospitals.

Mehra says he and other co-authors asked for an independent audit of the data in both papers from Medical Technology & Practice Patterns Institute (MTPPI), a health-care data research institute in Bethesda, Maryland. But on 3 June, Mehra says, MTPPI told them that Surgisphere would not share the data “because of agreements with its clients and the fact that the documents contain confidential information”. Mehra and his co-authors asked for retractions.

Before they published the studies, researchers and journals should have asked more questions about how so comprehensive a data set had been gathered from hospitals across the world in the middle of a pandemic, says Wendy Rogers, a bioethicist at Macquarie University in Sydney. “It is always easier with hindsight,” she says, “but in general with the COVID-19 research, there has been such a rush that all sorts of really abysmal papers are getting published.”

Malaria drug confusion

Researchers testing hydroxychloroquine in large clinical trials say they’re worried that the publicity around the Lancet findings may make it harder to complete their research, even though the paper has now been retracted. “We’re hearing that people just aren’t interested in hydroxychloroquine,” says David Smith, an infectious disease specialist at the University of California, San Diego, who is helping to run a trial funded by the US National Institute of Allergy and Infectious Diseases to test the drug in people with COVID-19 who have not been hospitalized. “The retraction won’t get anywhere near as much news as the original study,” he says. “We may never get an answer about treatment with hydroxychloroquine.”

Most data on hydroxychloroquine in COVID-19 has come from in vitro studies or small clinical trials. On 5 June, however, researchers with a large randomized UK trial, called RECOVERY, announced that their data on more than 4,600 hospitalized patients indicated that hydroxychloroquine didn't reduce the risk of death, and they were stopping that arm of the trial. And a randomized, controlled study in 820 people reported on 3 June that the drug was not effective in preventing infection among people who were at high risk of infection following exposure to someone with COVID-19³. In contrast to the retracted Lancet paper, the studies found no evidence of toxicity to the heart from hydroxychloroquine treatment.

Those findings don’t rule out possible benefits from hydroxychloroquine in other settings, says Joseph Cheriyan, a clinical pharmacologist at the Cambridge University Hospitals NHS Foundation Trust in the United Kingdom, where trials testing the drug were halted pending a safety review after the Lancet paper. (One has since resumed.) “The only way we’re going to know is if we do the trials, and what is frustrating is that this article has essentially delayed us,” he says.

It is not uncommon for studies that use large data sets to be published without external scrutiny of the raw data, says Smith. But an exception to this is when the paper is expected to have particularly high impact, as in the Lancet case. In this instance, he says, it appears that extra-careful review was skipped. “It’s a rush right now,” he says. “We are desperate for knowledge, and maybe we’re skipping over some of our tried-and-true checks.”

Both The Lancet and the NEJM said their peer review processes were confidential, and that they could not give details about how quickly the papers were reviewed. “The article had external peer review and statistical review, as well as scientific and manuscript editing. However, as we would do with any incident of this type, we are reviewing our procedures, including how we assess research analyzing large datasets based on electronic medical record data," said an NEJM spokesperson.

Ivermectin effect

Surgisphere’s ivermectin preprint may have a longer life. The Peruvian government included the drug in its national treatment guidelines a few days after a white paper cited the SSRN preprint, Chaccour says, and a week after that, Bolivia added ivermectin to its treatment guidelines, citing the situation in Peru. Paraguay has issued restrictions on the sale of ivermectin because of high demand.

After Chaccour gave a talk about ivermectin to the Peruvian Academy of Sciences, streamed on Facebook, he says he gets texts almost every day from people saying that they have an infected family member and can obtain formulations of ivermectin intended for use in animals, not people. “They send me a photo of a veterinary formulation and ask me, ‘Would you mind telling me what dose we should use?’” he says. “It is broadly available and people are broadly desperate and this could lead to misuse.”

Because the two versions of the preprint were not published in a journal, there is no retraction mechanism to limit their long reach, Chaccour worries. “Who retracts this ivermectin ghost in Latin America?” he says. “There’s no high-profile journal saying this was wrong.”