The Limit of Detection Matters: The Case for Benchmarking Severe Acute Respiratory Syndrome Coronavirus 2 Testing

Abstract Background Resolving the coronavirus disease 2019 (COVID-19) pandemic requires diagnostic testing to determine which individuals are infected with severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2). The current gold standard is to perform reverse-transcription polymerase chain reaction (PCR) on nasopharyngeal samples. Best-in-class assays demonstrate a limit of detection (LoD) of approximately 100 copies of viral RNA per milliliter of transport media. However, LoDs of currently approved assays vary over 10,000-fold. Assays with higher LoDs will miss infected patients. However, the relative clinical sensitivity of these assays remains unknown. Methods Here we model the clinical sensitivities of assays based on their LoD. Cycle threshold (Ct) values were obtained from 4700 first-time positive patients using the Abbott RealTime SARS-CoV-2 Emergency Use Authorization test. We derived viral loads from Ct based on PCR principles and empiric analysis. A sliding scale relationship for predicting clinical sensitivity was developed from analysis of viral load distribution relative to assay LoD. Results Ct values were reliably repeatable over short time testing windows, providing support for use as a tool to estimate viral load. Viral load was found to be relatively evenly distributed across log10 bins of incremental viral load. Based on these data, each 10-fold increase in LoD is expected to lower assay sensitivity by approximately 13%. Conclusions The assay LoD meaningfully impacts clinical performance of SARS-CoV-2 tests. The highest LoDs on the market will miss a majority of infected patients. Assays should therefore be benchmarked against a universal standard to allow cross-comparison of SARS-CoV-2 detection methods.


Supplementary Methods: Conversion from Ct to Viral Load
Derivation. RT-PCR depends on the exponential amplification of template to form product, which is measured using a fluorescence signal that is directly proportional to the concentration of product (e.g. an intercalating fluorophore). Fluorescence signal is monitored over time as repeat cycles of PCR are performed. In the early cycles, the signal is typically below the detection threshold and so appears as a flat baseline; this is the lag phase. As exponential amplification continues, the signal crosses the detection threshold and exponential growth becomes apparent; this is the log phase. As amplification continues further, formation of double-stranded product and primer consumption inhibit the amplification reaction [1,2], causing the growth of the signal to stall; this is the plateau phase. Typically a signal threshold ( ) is chosen within the region where the log phase is apparent; the cycle at which the signal crosses is called the Ct value. Note that because monitoring is continuous, the Ct value can be fractional (e.g. 6.54 cycles, as opposed to cycle 6 or cycle 7).
To convert from Ct value to viral load requires an expression that relates the two. To derive this expression, we first consider the relationship between signal intensity (y) and cycle number (x), which as described above is exponential: Here y0 is the intensity of the input material (generally far too low to measure directly). Recall that intensity is directly proportional to viral load in the sample (v); we can therefore substitute y = Cv, where C is a proportionality constant in units of fluorescence intensity units per copies/mL: We use e k , where k denotes the rate of growth, for ease of illustrating the exponential form of this expression: in every cycle, the amount of product at the end of the cycle is e k times the amount at the start of the cycle; thus e k is a ratio. Because PCR results in a doubling of product in each cycle (at maximum efficiency and ignoring measurement error, inhibitors, and other extenuating factors), we know that e k will have a value in the vicinity of 2. However we also know that inhibition of the polymerase by PCR product will make this ratio fall with each cycle, and we will need to measure this fall or else risk underestimating the amount of starting material. Therefore it is useful to make a substitution for e k , to the ratio ρ, yielding and because ρ is expected to vary (fall) with x, to write ρ as a function of x: We can measure ρ(x) for a given PCR reaction from the signal-vs-cycle plot. In the interval x1 to x2, the amount of signal will increase from y1 to y2 by some multiple that depends on ρ: Because ρ is what we desire to measure, we solve for it, yielding We acquired screenshots of the signal-vs-cycle curve for 50 randomly chosen positive samples and extracted the data using WebPlotDigitizer [3]. This yielded ~200 datapoints for each sample at a density of 7-30 datapoints per PCR cycle. We then used Eq. 3 to measure ρ at every datapoint. To minimize sampling error, we actually measured ρ on 20 x1-x2 intervals centered around each datapoint (consecutive datapoints, next-nearest neighbors, and so on), binned to every 1/10 th of a cycle, and took the median ρ for each bin, resulting in smooth curves of efficiency (=ρ-1) vs. cycle number (e.g. Fig S1).
As expected, following a short interval during which efficiency appeared to increase as a result of signal crossing the detection threshold and becoming quantifiable, efficiency peaked and then fell with cycle number. The peak efficiency exceeded 1 for several samples, as is possible from literature on RT-PCR, which mentions measured efficiencies up to 1.3 (ρ up to 2.3) [4]. The peak efficiency reliably occurred 1.41±0.93 (mean±stdev) cycles after the machine-reported Ct, a not surprising finding, since the Abbott Ct or FCN is based on modeling of peak efficiency, i.e., the cycle number at the so called maxRatio [5]. Also as expected, there was a negative association across samples between efficiency and Ct number, well fit (R 2 =0.82) by a linear relationship with Theil-Sen slope m=-0.028/cycle and intercept b=1.34 (Fig. S2). This relationship provides a measure of ρ(x) from Eq. 2, or more precisely, it provides ρ(Ct), which is needed to solve for v0 as a function of Ct.
Using ρ(Ct) to account for the decreasing efficiency with Ct, the signal threshold that corresponds to Ct can be approximated as where now the sum is from x=1 to x=Ct-1. Because the data we have gives fluorescence at fractional cycles, it is useful to convert from discrete (here, per-cycle) growth to continuous growth: with the same limits as above. Integrating and evaluating at these limits and simplifying slightly: From the manufacturer and our validation, we know that at the LoD v0=100 copies/mL (vL) and Ct=26.06 (CtL). Therefore: log = logC + logvL + (b/m+CtL-1) ×log(m(CtL-1)+b) -(b/m+1) ×log(m+b) -CtL + 2 Eq. 5 Subtracting Eq. 5 -Eq. 4 and moving logv0 to the left-hand side: main text), a standard quantified in viral genome copies/mL by droplet digital PCR, and was determined to be 50 genome copies/mL by simple logistic regression. Therefore, we assigned a value of 50 genome copy/mL to the serial dilution and cycle threshold of the FDA reference standard at the assay LoD. The Ct value for the next 10-fold more concentrated reference standard dilution was assigned a value of 500 copies/mL, etc.
The Ct values predicted by the model and obtained from the reference material at each serial log10 titer are plotted in Fig. S3. The concentration of the available reference material only allowed replicate tested at the five log10 dilutions spanning 50 to 5 x 10 5 genome copies/mL. However, within this range, the model and calibrator show almost an identical slope and nearly complete overlap. Therefore, we conclude that Eq. 6 is highly predictive of the relationship between Ct and viral load. We expect that significant deviations from model predictions are less likely to occur at very early cycle thresholds, i.e., greater than 10 6 genome copies/mL, as PCR reaction inhibition observed at high cycles numbers is limited and therefore amplification parameters are more predictable. Therefore, we predict that the extension of model predictions to 10 9 genome copies/mL is likely to be reasonably accurate.