Concerns About the Special Article on Hydroxychloroquine and Azithromycin in High-Risk Outpatients With COVID-19

Abstract

In May 2020, the Journal published an opinion piece by a member of the Editorial Board, in which the author reviewed several papers and argued that using hydroxychloroquine (HCQ) + azithromycin (AZ) early to treat symptomatic coronavirus disease 2019 (COVID-19) cases in high-risk patients should be broadly applied. As members of the Journal’s Editorial Board, we are strongly supportive of open debate in science, which is essential even on highly contentious issues. However, we must also be thorough in our examination of the facts and open to changing our minds when new information arises. In this commentary, we document several important errors in the manuscript, review the literature presented, and demonstrate why it is not of sufficient quality to support scale up of HCQ + AZ, and then discuss the literature that has been generated since the publication, which also does not support use of this therapy. Unfortunately, the current scientific evidence does not support HCQ + AZ as an effective treatment for COVID-19, if it ever did, and even suggests many risks. Continuing to push the view that it is an essential treatment in the face of this evidence is irresponsible and harmful to the many people already suffering from infection.

azithromycin, bias, confounding, hydroxychloroquine, observational studies, randomized trials

Abbreviations

 
• AZ

  azithromycin

 
• CI

  confidence interval

 
• COVID-19

  coronavirus 2019

 
• HCQ

  hydroxychloroquine

 
• RCT

  randomized controlled trial

Editor’s Note: The opinions expressed in this article are those of the authors and do not necessarily reflect the views of the American Journal of Epidemiology.

In May 2020, the Journal published an opinion piece by a member of the Editorial Board, Dr. Harvey Risch (1), in which he reviewed several papers—published and unpublished—and argued that using hydroxychloroquine (HCQ) + azithromycin (AZ) early to treat symptomatic coronavirus disease 2019 (COVID-19) cases in high-risk patients early should be broadly applied. The article received little attention until the author wrote a similar piece for Newsweek (2), which was followed by interviews on Fox News, implying that by publishing his opinion, the American Journal of Epidemiology Editorial Board agreed with his assessment. This is categorically not the case.

As members of the Journal’s editorial board, we are strongly supportive of open debate in science, which is essential even on highly contentious issues. However, we must also be thorough in our examination of the facts and open to changing our minds when new information arises. Although the article in question is an opinion piece, not a systematic scientific review of the evidence, that does not excuse inaccurate reporting of the facts. The article contains numerous factual errors and ambiguous statements that need to be corrected or clarified. These issues, which we detail in the present article, are not open to interpretation; rather, they are a clear misstatement of the evidence from the cited materials or misstatements of epidemiologic principles. Furthermore, since Dr. Risch’s article was published, new information has come to light that must be taken into consideration when formulating an opinion on the efficaciousness of a treatment, and Dr. Risch’s position on use of HCQ + AZ must be evaluated in the context of this new, unfavorable, information.

This occurrence, like many others during this time of COVID-19, has laid bare longstanding tensions among the areas of scientific endeavor, medicine, and public health practice, and between the private good and the public good. However, the urge for swift medical action and to do the best by patients does not give us license to ignore evidence or fail to update our views when that evidence changes, because doing so hurts the patients themselves and public health as a whole.

INACCURACIES AND ERRORS

Dr. Risch identified 5 studies (3–8) that he used to argue that HCQ + AZ is an effective early treatment for COVID-19 in high-risk outpatients (1). We deal with the quality of the evidence in these studies in the next section but first want to detail a number of inaccuracies in Dr. Risch’s presentation of the evidence that we feel need to be corrected. This is not a comprehensive list but rather some key points we think are essential to his argument.

1. In summarizing the study by Gautret et al. (3), Dr. Risch referred to a 50-fold benefit of HCQ + AZT versus the control, but nowhere in the Gautret et al. article is a 50-fold increase reported. In the cited re-analysis (9), there is a reported odds ratio of 52, but given the outcome is so common, the odds ratio will be a markedly inflated version of the more relevant parameter, the risk ratio. Using the data from Gautret et al. (3), we calculate a risk ratio of approximately 5. We also note no confidence intervals were presented in Dr. Risch’s article (1), so there is no way to see if the data are consistent with very large and very small associations. This is particularly important given the small size of this study (just 42 patients total, subdivided into 3 groups; some patients were excluded). Small studies in which large effects are found, even if unbiased and from trials, tend to overstate effect sizes (10). In the re-analysis (11) cited by Risch, a risk ratio of 3.84 was reported for HCQ overall, and no results were reported for the risk ratio for HCQ + AZ, because of model separation. When switching to the odds ratio, we see that the confidence interval for the 52-fold odds ratio goes from 1.95 to 1,399, an incredibly imprecise finding. Thus, even if the effect is real, it is very likely overstated.

2. Dr. Risch, in referring to the study by Gautret et al. (3), made the argument that a large effect cannot be explained by confounding. This is incorrect. Large effects can be explained by confounding, particularly when the relationships between the omitted variable and the variables of interest are strong (think of the classic example of the relationship between carrying matches and lung cancer) or the sample size is small. In addition, selection bias, measurement error, and other sources of bias mean it is very plausible that the observed association could be explained by bias and random error.

3. Referring to the Gautret et al. study (3), Dr. Risch then stated, “Further, the study showed a significant, 7-fold benefit of taking HCQ + AZ over HCQ alone (P = .035), which cannot be explained by differential characteristics of the controls, since it compares one treatment group to the other” (1, p. 1220). This statement suffers from the same issues outlined in key points 1 and 2, that is, no description of the outcome or effect measure, lack of confidence interval, and very small sample size. In addition, the statement that this cannot be explained by “differential characteristics of the controls” does not make sense, because with such a small (nonrandomized) sample, it is entirely plausible that there are differential characteristics among the 14 patients who received HCQ alone and the 6 who received HCQ + AZ. Furthermore, even in a large, nonrandomized study, there can be important differential characteristics between those who received HCQ alone and those who received HCQ and AZ.

THE EVIDENCE FOR THIS TREATMENT, EVEN IN MAY 2020, WAS WEAK AND OF LOW QUALITY

Leaving aside the issue of errors and inaccuracies in his opinion piece, we still strongly disagree with the position Dr. Risch has taken on HCQ + AZ. We believe that the evidence, even in May 2020 when the piece was written, did not support the use of these drugs, even for early outpatient treatment of COVID-19 for high-risk patients. There have been many critiques of Dr. Risch’s article and the individual studies he cited; as examples, see comments by Rosendaal (12) and Machiels et al. (13), among others, on the Gautret et al. study (3). There have also been numerous letters published in response to Dr. Risch’s article (1); we support the responses of those authors (14–16). We do not wish to reproduce all of these critiques and so here we restrict ourselves to some additional key points.

As noted, Dr. Risch considered 5 studies when he evaluated the evidence. Three of these had no control groups (5–8) and, therefore, cannot be used as evidence for estimating the causal effect of HCQ + AZ on COVID-19 outcomes. This leaves only 2 studies from which to draw conclusions about the risks or benefits of this treatment regimen (3, 4). Both of these studies have serious limitations we consider so severe that they provide little evidence.

The Gautret et al. study (3) was a small study. The small sample size alone provides cause for concern, despite Dr. Risch’s assertion to the contrary (1). Even a randomized trial of 42 patients (6 of whom were lost to follow-up) in which only 6 patients received the treatment of interest would provide very weak evidence for an effect. But this was not a randomized trial. Yet, it appears no measures were taken to control for confounding at either the study design or the analysis phase. Few data on potential confounders were even reported to allow for such adjustment to be done. As such, this study, at best, could provide exploratory evidence suggesting this treatment should be examined further. It cannot provide sufficient evidence on which to base policy, despite the urgency for action. This limitation became even more problematic when Dr. Risch attempted to stratify the already thin data into those who did and did not have “presentation with asymptomatic or upper respiratory tract infection, versus lower respiratory tract infection” (1, p. 1220) to make the case that the medication needs to be used as early as possible in treatment. Given the small sample size, the data cannot support stratified analysis.

The second study, by Barbosa et al. (4), was an observational study conducted in Brazil. It was larger than the Gautret et al. study (3), having 412 patients who received the treatment. However, in this case, the authors used as control participants the 224 patients who refused treatment. In such a case, confounding is all but assured. It is very likely the groups differed with respect to the risk for outcomes. This confounding, if not corrected for, means we cannot learn much from the study. Without detailed information about the participants of the trial, it is impossible to determine the impact of this confounding on the results. For example, if the sickest patients agreed to take HCQ + AZ, then confounding by indication could make this treatment regimen look less beneficial. However, it is important to remember that this study (and Dr. Risch’s recommendations) focused on outpatient treatment of patients early in the COVID-19 pandemic who had mild symptoms. Therefore, Dr. Risch’s argument that those who chose to try the experimental treatment were sicker (and thus presumably desperate to try anything) seems unsupported. A plausible alternative is that the sicker patients decided to decline this treatment in favor of seeking more intensive care elsewhere. This would supplement the control group with individuals who were the most likely to die and overinflate the observed benefits of the treatment regimen. Furthermore, there appears to have been no confirmation of infection with severe acute respiratory syndrome coronavirus 2 in this study, making it impossible to determine effectiveness of the treatment for the condition specified. The direction of confounding by indication can be difficult to predict and can go in either direction.

The final 3 studies included by Dr. Risch (5–8) were case series with no control group. Such studies can be useful for prompting new research questions and guiding future research; they cannot be used to support a causal conclusion about the efficacy of a treatment. Although universally fatal conditions such as rabies and some cancers represent a rare exception to this rule, COVID-19 in no way meets this criterion. Hence, these studies do not support Dr. Risch’s argument. Given these severe limitations, we do not consider any of these studies useful for policy making.

THE EVIDENCE HAS MOVED ON

Dr. Risch noted, “Each piece of evidence, contained in each study, must be carefully considered and not dismissed, because in an ideal world such evidence would fall in a lower part of the evidence-quality triangle.” (1, p. 1220). We agree with this, but that is also not a reason to accept flawed research and use it to make strong statements that the evidence cannot support (indeed, the title of Dr. Risch’s article calls for “ramping up” this treatment). As Dr. Risch noted, all treatments have risks (1) (he falsely noted that they all have benefits, which is not the case); as such, using bad evidence can lead to harm. In addition, if one is going to act on such evidence, it is critical that they continue to collect additional information and reevaluate their conclusions as more evidence is generated. In this case, since the publication of his article, stronger randomized trial evidence has emerged. Dr. Risch has dismissed these studies because he does not believe evidence from randomized controlled trials (RCTs) is, de facto, more valuable than observational data. This is a straw man argument; we, as fellow epidemiologists, also believe strongly that observational evidence should be given due consideration, but as a general rule, well-conducted RCTs give a more accurate estimate of a causal effect than poorly designed, small observational studies. In fact, in a large observational study of treatment with HCQ alone or with zinc, compared with neither, researchers did find a protective association with death for each treatment (interestingly finding approximately the same association for both), but they rightly concluded, “Prospective trials are needed to examine this impact” (17, p 396).

We note that Dr. Risch, in his own response to letters of concern, presented 7 additional studies that were published since his May 2020 letter (11). Four of the 7, again, had no control population and cannot be regarded as evidence. The remaining studies, except for 1, were listed as personal communications and, as such, cannot be evaluated, and the remaining had no control group and was a media report. None were RCTs, which would provide much stronger evidence because they would be better able to limit the impact of confounding.

In May 2020, around the time of Dr. Risch’s comment, a report of an RCT was published in which researchers did compare HCQ (not with AZ) with the standard of care in adults in China (18). These were hospitalized patients, but almost all (n = 148 of 150) had mild to moderate coronavirus infection and the outcome was negative conversion between 2 and 28 days. Perhaps a small benefit was seen, but with very poor precision (risk difference, 4.1%; 95% confidence interval (CI): –10.3%, 18.5%; the authors concluded no benefit on the basis of the P value) (18). Dr. Risch has argued that we cannot use studies in which HCQ and AZ were not used as evidence and, as noted, this study in China did not include AZ. However, the study findings do provide more evidence against any large benefits such as those seen in the Gautret et al. (3) study. A similar result was found in a trial of HCQ (again, not with AZ), by Skipper et al. (19), of 423 adult outpatients in the United States with early, mild disease, 341 of whom had laboratory-confirmed infection. The authors found very little difference in symptom severity compared with placebo at day 14 (–0.27 points; 95% CI: –0.61, 0.07 on a 10-point scale). In a third trial of HCQ alone in Spain of early treatment among 293 patients with mild disease, authors Mitjà et al. (20) found no differences in mean reduction of viral load at day 3. There was some reduction in risk of hospitalization (risk ratio = 0.75; 95% CI: 0.32, 1.77), but the authors concluded no difference on the basis of the lack of significance.

A more relevant study, a randomized trial from a group in Brazil, has been published by Cavalcanti et al. (21). They randomly assigned 667 patients with mild to moderate disease to either HCQ + AZ, HCQ alone, or standard of care. Of those, 504 were then confirmed to have had COVID-19, and among those, there was no benefit to HCQ + AZ on clinical status at 15 days (odds ratio = 0.99; 95% CI: 0.57, 1.73). Dr. Risch has argued that the treatment is only effective in high-risk patients, including older than 60 years and those with comorbidities (1). Although the study by Cavalcanti et al. (21) was not limited to those older than 60 years, the mean age was approximately 50 years. Approximately 39% of the participants had hypertension, approximately 16% were obese, and approximately 19% had diabetes. A subgroup analysis shown in the report’s appendix that was limited to those older than 60 years also showed no benefit (odds ratio = 0.95; 95% CI: 0.37, 2.43). Thus, although not limited to high-risk patients, clearly the high-risk patients Dr. Risch says are the ones that need to be studied made up a strong proportion of this study population.

No doubt these studies too have limitations, such as lack of blinding, but the evidence they provide is still much stronger than that presented by Dr. Risch. And certainly, the result of 1 trial is not enough to draw strong conclusions either, but to date, this is the best evidence we have, and it does not support use of HCQ + AZ in early disease. Given this new RCT evidence, there is no way to continue to support the position that HCQ + AZ use is beneficial, nor that it will save 100,000 lives unless some new evidence were to emerge that would support such a statement.

CONCLUSION

We believe that Dr. Risch’s summary of the evidence in May 2020 (1) contained factual errors, and that the evidence presented, even at the time of publication, was weak at best. Since then, stronger evidence has demonstrated no benefit for early HCQ + AZ treatment of high-risk patients with mild COVID-19. Disagreement and opposing points of view are welcome in science, and there is some subjectivity in deciding when a study’s flaws are sufficient to invalidate its conclusions. However, we do not believe that Dr. Risch provided a sound basis for refuting the scientific evidence against a benefit for early treatment with HCQ + AZ in high-risk patients with mild COVID-19, and he certainly did not provide evidence to support policy claims such as “ramping up” this treatment.

Dr. Risch made the comment, “In this context, we cannot afford the luxury of perfect knowledge” (1, p. 1220). We wholeheartedly share Dr. Risch’s desire for an effective treatment in the face of this pandemic, and we would be ecstatic if the evidence supported HCQ + AZ or any other treatment. However, when acting on limited evidence, we must be careful not to view the data with rose-colored glasses, and we must be quick to adjust our views, abandoning positions as the evidence changes. Unfortunately, the current scientific evidence does not support HCQ + AZ as an effective treatment for COVID-19, if it ever did. Continuing to push the view that it is an essential treatment in the face of this evidence is irresponsible and harmful to the many people already suffering from infection.

ACKNOWLEDGMENTS

Author affiliations: Department of Epidemiology, Boston University School of Public Health, Boston University, Boston, Massachusetts, United States (Matthew P. Fox, Eleanor Murray); Department of Mathematics and Statistics, Wake Forest University, Winston-Salem, North Carolina, United States (Lucy D’Agostino McGowan); Department of Medicine, Section of Epidemiology Research, Rush University Medical Center, Chicago, Illinois, United States (Bryan James); and Department of Epidemiology, Johns Hopkins Bloomberg School of Public Health, Johns Hopkins University, Baltimore, Maryland, United States (Justin Lessler, Shruti Mehta).

Conflicts of interest: In the past 5 years, M.P.F. has received speaking fees from Merck; L.D.M. has received consulting fees from Acelity and Sanofi; B.D.J. has received consulting fees from the Alzheimer’s Association; S.H.M. has received speaking fees from Gilead; and E.J.M. has received consulting fees from Blue Cross Blue Shield of Massachusetts.