The past few months have spurred an unprecedented scientific effort to understand and address COVID-19 and the virus that causes it. We hope that improvements in our understanding of the disease processes will quickly lead to novel therapeutic interventions and improved outcomes.
With the rapid generation of data comes the real problem of assessing the quality of the data. The typical protective mechanism, the scientific review process, is meant to ensure that only high-quality research is published. Although no system is perfect, the criticism and feedback provided by reviewers are critical to improving the science and furthering research. Unfortunately, the number of journals publishing primary research has proliferated to the point that, even with the journal hierarchy gauged by the impact factor, there is still ample opportunity for poor-quality research to be published. Additionally, the use of the lay press as a conduit of information enables researchers to bypass this process altogether.
As I see it, one of the greatest risks to our quality control process is the lack of open criticism and dissemination of that criticism with the data. I am not suggesting any misrepresentation or malfeasance. The authors typically use the discussion section to point to the strengths and weaknesses of their own work. The problem is that most of us read only the abstract, and never learn about the self-reported flaws inherent in the work. Most of us do not have the opportunity to debate the validity of the data, the methodology, and how to apply the data to clinical practice. Journal articles that do not describe their results with adequate caveats and criticism represent a true failing of our system.
The urgency associated with the COVID-19 pandemic has provided additional opportunities for the publication process to fail to prevent questionable data from being released, as exemplified by a recent report on hydroxychloroquine. I would like to preface this discussion by assuming that the researchers were making a good faith effort to test a novel treatment for a deadly infection. On March 20, a report by Gautret and colleagues appeared online in the International Journal of Antimicrobial Agents on the use of hydroxychloroquine and azithromycin as treatment for COVID-19. The authors concluded that their study, although small, showed that “hydroxychloroquine treatment is significantly associated with viral load reduction/disappearance in COVID-19 patients and its effect is reinforced by azithromycin.” This report was then followed on March 21 by a tweet by President Trump indicating that when taken together, hydroxychloroquine and azithromycin “have a real chance to be one of the biggest game changers in the history of medicine.”
Unfortunately, the published paper was highly flawed. The data do look impressive, with 14 of 20 patients (70%) in the treatment arm achieving viral clearance vs 2 of 16 (12.5%) in the control arm. But the analysis was cursory and nonrigorous, with findings that were suitable only for hypothesis generation and the undertaking of prospective, randomized controlled trials. Instead, the findings were heralded as fact and put into clinical use.
A complete analysis of the available data by Dahly and colleagues (doi:10.5281/zenodo.3725560) describes the statistical and methodologic flaws in the research. I encourage readers to review this analysis, which provides an education in scientific methodology and helps enable a better interpretation of the data. In brief, the study lacked randomization, used inappropriate controls (including patients who elected to not participate in the study), discarded patients because of incomplete data, selected an endpoint with unclear clinical value, dichotomized a continuous variable, and did not use azithromycin uniformly among the patients. Others, including Kim and colleagues in the Annals of Internal Medicine on March 30, have pointed to additional issues, most importantly the lower baseline levels of the virus in the hydroxychloroquine/azithromycin group, suggesting that these patients were further along in their disease course than those in the control group.
The report by Gautret and colleagues, combined with President Trump’s tweet and reporting by the lay press, led to a 1977.0% increase in prescriptions in the United States for hydroxychloroquine/chloroquine compared with the same week in 2019 (Vaduganathan and colleagues in the Journal of the American Medical Association, posted May 28). We would not expect laypeople to have the acumen to critically review the literature before requesting prescriptions from their physicians. However, physicians who had the skill set to interpret the data did write the prescriptions. So where does the problem lie? There is no one correct answer, although I have my own opinions. I would be interested in knowing the opinions of others as well (please send your thoughts to info@clinicaladvances.com). The goal is not to curtail investigation nor the dissemination of data, but to make sure that—whether we are treating patients an ancient disease like cancer or a novel one like COVID-19—we continue to “do no harm.”
Sincerely,
Richard R. Furman, MD