Unraveling the flaws of estimates of the infection fatality rate for COVID-19

COVID-19, Infection Fatality Rate

The infection fatality rate (IFR) of COVID-19 is one of the measures of disease impact that can be of importance for policy making. However, we have found that there are significant flaws in many of the studies that attempt to estimate the IFR, which lead to biased results, potentially affecting public discourse and policy actions.

In order to demonstrate some of the flaws, we consider papers cited in a meta-analysis² that the CDC had used as the basis for its estimate of IFR for an extended period of time.¹ We have previously written a criticism of one of the 24 reports used by the meta-analysis, which was based on data from the Diamond Princess cruise.^3,4 Starting from an incorrectly estimated 1.3% IFR for the Diamond Princess—the actual IFR for the Diamond Princess being 2.0%—they attempted to model the IFR in China and reported a value 0.6%; however, the multiple errors in their calculations, including dividing by the same factor twice, render their results meaningless. Here, we discuss flaws in five additional studies cited in the meta-analysis. We note that these studies were not selected because of the presence of flaws, but instead they were sequentially chosen in each of the three categories in the pre-print version of the meta-analysis. We discuss two types of problems: flaws in the studies per se and flaws in the way the meta-analysis used them.

The Rate of Underascertainment of Novel Coronavirus (2019-nCoV) Infection: Estimation Using Japanese Passengers Data on Evacuation Flights

At the end of January, Japan conducted three repatriation flights of Japanese nationals and their families on 29–31 January from Wuhan, China.⁵ Upon disembarkation from the planes, tests were performed of the passengers and the test results were used to estimate the IFR in Wuhan

Non-representative sample

Japanese nationals likely differ in ways relevant to disease transmission due to their social networks and the geographic areas they frequent, and the 565 passengers are not independent samples due to the presence of family units and correlated social networks and behaviors.

Nonsensical mathematical analysis

The authors write an equation in which they set the cumulative number of infections in Wuhan c(t)/q (the number of detected infections divided by the ascertainment rate) × the detection window |$T$| divided by the total population of Wuhan |$n$| equal to the fraction of evacuees who tested positive (⁠|$8/565$|⁠)—i.e.

$$ \frac{8}{565}=\frac{c(t)T}{qn}. $$

However, the left-hand side of the equation is dimensionless while the right-hand side has units of time. This renders the equation meaningless.

No external validity to the ascertainment rate

Even if the ascertainment rate were correctly derived, however, it has no validity for application to other contexts, as it applies only to a particular snapshot in time when tests in China were just starting to ramp up. Any application of this rate to convert CFR to IFR under other conditions is, therefore, inappropriate and misleading (see below).

Ignoring confidence interval uncertainty

The authors note that their estimate of q = 10% has a 95% confidence interval of 5.0% to 20%. Yet, when subsequently calculating the error range of IFR|$=$| CFR |$\times q$|⁠, they use 10% for |$q$| without accounting for its confidence interval. Of course, given all of the flaws present in deriving the ascertainment rate, even a correctly calculated error range would be meaningless.

Real-time Estimation of the Risk of Death from Novel Coronavirus (COVID-19) Infection: Inference Using Exported Cases

This study modeled the epidemic growth rate in China and estimated a CFR of |$5\hbox{--} 8\%$| and an IFR of |$0.5\hbox{--} 0.8\%$| by multiplying the CFR of the study by the ascertainment rate from the previously described study.⁶

Inappropriate duplicate use of studies

There is no independent assessment of the ascertainment rate.⁶ Since the ascertainment rate is a key determining factor for IFR, this is not an independent evaluation of the IFR as would be required for correct inclusion in a meta-analysis.

Flawed ascertainment rate

We note again that the ascertainment rate that is being used has multiple invalidating flaws detailed previously.

Estimating the Burden of SARS-CoV-2 in France

By assuming that the French and Diamond Princess cruise ship populations differ only in their age/sex composition and by assuming a particular distribution of infected individuals across age/sex groups, the authors estimate the IFR for each age/sex group in France.⁷ They report an overall IFR of 0.5% (95% credible interval: 0.3–0.9%).

Misstatement of study’s conclusions

In the abstract, the authors state that ‘0.5% of those infected die’ but later in their manuscript they note that all COVID-19 deaths occurring outside of hospitals—and in particular all retirement/nursing home populations—are excluded from their analyses. While it is valid to conduct an analysis that applies only to part of a population, it is misleading to report results derived from such analyses as if they apply to the population as a whole as the authors do in the abstract, particularly given that they end up excluding 9604 of the total 25 990 COVID-19 deaths reported by May 7 (the final date for data used in this study).⁷ If hospitalized deaths are included, it increases their IFR estimates by a factor of approximately 1.59.

Unjustified assumptions regarding relative probabilities of infection

They assume that infections for each age/sex group are proportional to contact rate based on the supposed linearity of their Fig. S17. However, Fig. S17 actually has substantial curvature to it, bringing this assumption into question. Furthermore, the data they use to ascertain the contact rates of each of the age/sex groups is from a study conducted in 2012 rather than during the pandemic period. The authors then make a series of assumptions that are not validated by empirical data as to the effect of the lockdown on these contact rates in each age group.

Non-representative sample (Diamond Princess) used to set the overall IFR

The IFRs by age/sex group that they calculate are essentially derived from those on the Diamond Princess, using the French data to understand the relative effect of age/sex on IFR. Their analysis is accurate only to the extent that the IFR for each age/sex group on the Diamond Princess equals the IFR for that age/sex group in France. This assumption is not justified; individuals on the Diamond Princess could substantially differ from individuals of the same age and sex in France. In particular, cruise ship passengers are, for their age, likely to be healthier and (due to active screening) to have received more rapid medical attention than the general population. Their sensitivity analysis in which the Diamond Princess population has a 25% lower IFR for each age/sex group is inadequate; it would not be surprising if the Diamond Princess IFRs for each particular age/sex group differed from their French counterparts by a greater factor.

No justification of sensitivity analysis for Bayesian priors

Their paper is based on Bayesian analysis, but they provide no justification for their choice of priors, nor is any sensitivity analysis for the priors reported. It is, therefore, unclear how their results would differ had they chosen a different set of priors.

Duplicate use of Diamond Princess data in meta-analysis

By including both the Diamond Princess study,⁴ and this study that estimated IFR based on the Diamond Princess data, the meta-analysis effectively includes IFR estimation of the Diamond Princess data twice, a similar meta-analysis error as found in Real-time Estimation of the Risk of Death from Novel Coronavirus (COVID-19) Infection: as found in the previous section.

Characteristics of COVID-19 Infection in Beijing

This is an observational study of patients from Beijing, with 262 confirmed cases that were transferred to designated hospitals and three deaths⁸. It is used as 1.15% IFR in the meta-analysis.

Non-representative population sample

The patients over-represent travelers (from Wuhan), a group that is typically younger and healthier than the general population.

Meta-analysis flaw

The study reports a CFR but the meta-analysis² misstates the CFR as the IFR.

COVID-19 Antibody Seroprevalence in Santa Clara County, California

This widely criticized study estimated a 2.8% prevalence rate, meaning infections were more than 50 times higher than the confirmed cases in Santa Clara County at the time of the study.⁹ The estimated IFR was 0.17%, one of the lowest IFR estimations the meta-analysis cited.

Non-representative sample

The population sample was obtained by online sign-up without measures to determine whether or not it was a representative sample.

Incorrect statistical accounting of false positives

There were only 50 positive test results out of a sample of 3330, a number that could be dominated by noise from even a small false-positive rate of the antibody tests.

Final Notes

Our review points out multiple flaws in a selection of papers used to estimate IFR. In addition to these specific flaws, none of the studies discussed above attempted to estimate the number of deaths from COVID-19 that were not attributed to COVID-19, leading to systematic underestimates of the IFR, as noted in the meta-analysis.² Excess mortality data¹⁰ suggest a significant number of unattributed deaths from COVID-19, although some of the excess mortality may be from indirect effects of the pandemic (which are nevertheless an impact of the pandemic).

It is also important to note that the IFR is not a biological invariant but rather significantly varies with the distribution of age, health and other qualities of those infected as well as the medical care they receive. Thus, the IFR calculated for a particular population of infected individuals is not immediately generalizable to other populations. Meta-analyses that consider IFR as a well-defined quantity and that average values from various populations to produce a single estimate without accounting for such differences are therefore problematic. The lead author has recently posted an additional meta-analysis that investigates age-stratified IFRs¹¹; such an analysis is an improvement but still does not take into account other factors such as differences in the underlying health of each age group and differences in access to treatment.

Authors contribution

The authors contribute equally to this work.

Funding

There is no funding for this submission.

Conflict of interest: The authors have no conflict of interest.