Abstract

Background

Next-generation sequencing (NGS) is gradually replacing Sanger sequencing (SS) as the primary method for HIV genotypic resistance testing. However, there are limited systematic data on comparability of these methods in a clinical setting for the presence of low-abundance drug resistance mutations (DRMs) and their dependency on the variant-calling thresholds.

Methods

To compare the HIV-DRMs detected by SS and NGS, we included participants enrolled in the Swiss HIV Cohort Study (SHCS) with SS and NGS sequences available with sample collection dates ≤7 days apart. We tested for the presence of HIV-DRMs and compared the agreement between SS and NGS at different variant-calling thresholds.

Results

We included 594 pairs of SS and NGS from 527 SHCS participants. Males accounted for 80.5% of the participants, 76.3% were ART naive at sample collection and 78.1% of the sequences were subtype B. Overall, we observed a good agreement (Cohen’s kappa >0.80) for HIV-DRMs for variant-calling thresholds ≥5%. We observed an increase in low-abundance HIV-DRMs detected at lower thresholds [28/417 (6.7%) at 10%–25% to 293/812 (36.1%) at 1%–2% threshold]. However, such low-abundance HIV-DRMs were overrepresented in ART-naive participants and were in most cases not detected in previously sampled sequences suggesting high sequencing error for thresholds <3%.

Conclusions

We found high concordance between SS and NGS but also a substantial number of low-abundance HIV-DRMs detected only by NGS at lower variant-calling thresholds. Our findings suggest that a substantial fraction of the low-abundance HIV-DRMs detected at thresholds <3% may represent sequencing errors and hence should not be overinterpreted in clinical practice.

Introduction

ART has led to a substantial reduction in HIV incidence and mortality.1,2 However, HIV may gain drug resistance mutations (DRMs) that reduce the effectiveness of ART and increase the risk of virological failure.3 Hence, it is important to test for HIV-DRMs to optimize the ART regimen, and it is recommended to perform genotypic drug resistance testing (GRT) before initiation of ART and at treatment failure.4–7

Sanger sequencing (SS) has been the gold standard test for GRT over the last two decades and has been shown to be highly reproducible and interpretable in clinical settings.8,9 SS can detect HIV-DRMs that appear in more than 15%–25% of variants.8–10 However, next-generation sequencing (NGS), particularly Illumina sequencing, is gradually replacing SS as the primary method for genotypic resistance testing.11–13 Several studies have shown that NGS is robust in detecting HIV-DRMs and may further detect low-abundance mutations.14–22 However, some of these studies were done on older NGS technologies such as Roche 454,14–16 and some were performed on test samples but not on patient samples.20,21 Most of these studies had low sample size (<100 samples) and they did not assess the dependency of low-abundance HIV-DRMs with variant-calling thresholds in detail (see Table S1, available as Supplementary data at JAC Online, for an overview of these studies).

In this study we compared, for participants enrolled in the Swiss HIV Cohort Study (SHCS), the HIV-DRMs detected by SS and NGS at different thresholds and explored the dependency of the frequency and spectrum of low-abundance HIV-DRMs on the variant-calling thresholds.

Materials and methods

Study population

Our study population consisted of people living with HIV (PLHIV) enrolled in the SHCS and the Zurich Primary HIV Cohort Study (ZPHI). The SHCS is a prospective multicentre study that is highly representative of the HIV-1 epidemic in Switzerland and includes broad, in-depth and high-quality genetic, biological, clinical and demographic data.23 As of August 2021, the SHCS has a cumulative total of 21 529 patients. Plasma samples from PLHIV in the SHCS are collected twice per year. The ZPHI is an observational, prospective, monocentre study at the University Hospital Zurich started in 2002.24 As of August 2021, more than 360 PLHIV with documented primary HIV infection had been enrolled in the ZPHI.

The SHCS Drug Resistance Database contains the HIV sequence data, primarily partial pol genes, sequenced from these plasma samples using SS. The SwissHIVGenomeDB is maintained by the Department of Infectious Diseases and Hospital Epidemiology at the University Hospital Zurich; it contains the HIV NGS data, predominantly (near) whole genome sequences, sequenced from plasma samples mostly by Illumina sequencing. SmaltAlign, an inbuilt pipeline in the SwissHIVGenomeDB, is developed by the Institute of Medical Virology, University of Zürich, and is used to iteratively align the reads of an NGS sequence against a given reference.25 Briefly, Illumina sample reads are read with seqtk, de novo assembly of sampled reads is performed with velvet, and iterative alignment of NGS reads and contigs is performed with Smalt. SmaltAlign reports the read depth and nucleotide frequencies at each position of the HIV genome for each NGS sample.

To compare the detection of HIV-DRMs by NGS and SS, we included all participants enrolled in the SHCS with SS and NGS sequences available for the pol region of HIV and with a difference in sample collection date for SS and NGS of 7 days or less (with the date requirement ensuring that genuine differences in frequencies due to within-host evolution should be minimal).

Sequencing

SS was performed by amplification and sequencing of HIV genomic regions targeted by antiretroviral (ARV) drugs, such as the protease (PR), reverse transcriptase (RT) and integrase (IN) regions. This method of population sequencing generates a single consensus sequence per sample, which represents the predominant HIV variant at each position or, using ambiguous nucleotides, minor variants with a frequency above 20%–30%.26 SS in the SHCS was performed by four laboratories using commercial and in-house methods.27

NGS sequences were generated using previously described protocols28,29 in the context of a variety of projects30–32 including the Bridging the Epidemiology and Evolution of HIV in Europe (BEEHIVE) collaboration.33,34 Briefly, the data were generated by manually extracting viral RNA from blood samples then reverse transcribing and amplifying the nucleic acids using primers to define four overlapping amplicons spanning the whole genome, which were fragmented and sequenced with Illumina MiSeq or HiSeq platforms.35 The sequences of the resulting ‘reads’ (nucleic acid fragments) were stored in the SwissHIVGenomeDB. We assembled these reads and obtained nucleotide frequencies at each position of the HIV genome for each NGS sample using the SmaltAlign pipeline.25 We then reconstructed the pol region of HIV genome for NGS sequences at different variant-calling thresholds.

Quality control

We defined a ‘sample pair’ to be an SS and NGS sequence pair available for the pol region of HIV for a patient enrolled in the SHCS and with a difference in sample collection dates for SS and NGS of 7 days or less. In the case of multiple SS and NGS sequences available, we considered the SS sequence that covers the highest number of nucleotides in the pol region and the NGS sequence with the highest median depth. We required sample pairs to meet the following criteria. First, that the SS sequence covered at least 250 nucleotides in the PR region, 500 nucleotides in the RT region or 500 nucleotides in the IN region for the respective region to be considered. Second, that the NGS sequence had a median depth of at least 100 reads per position. Third, we computed the pairwise distance between the nucleotide sequence of NGS at 20% variant-calling threshold and SS for each sample pair and excluded sample pairs with a pairwise distance ≥4.5%.

Drug resistance mutations and resistance level to ARV drugs

Both SS and NGS sequences were analysed using the Stanford HIV Drug Resistance Database genotypic resistance interpretation program (Algorithm version 9.0).36 For NGS sequences we reconstructed the pol region at different thresholds (1%, 2%, 3%, 5%, 10%, 15%, 20%, 25% and 30%). Resistance was compared either at the level of individual DRMs or at the level of the resistance score for the entire genotype. For the former case, we defined HIV-DRMs as those mutations that confer at least low-level resistance (DRM score ≥15) on their own (Table S2). For the latter case, we considered drug resistance level interpretations for the ARV drugs (Table S3). There were five predicted drug resistance interpretation levels: susceptible, potential low-level resistance, low-level resistance, intermediate resistance and high-level resistance. These levels were used to compare the detection of drug resistance levels detected by SS and NGS.

Sensitivity and additional analyses

We performed the following sensitivity analyses to assess the robustness and plausibility of our results. Firstly, we varied the distance thresholds used in the quality control. In the main analysis, we used a liberal threshold of DNA pairwise distance of 4.5% to select sample pairs with high quality, because sample pairs above this threshold could be due to mislabelling of a sample, technical errors in sequencing or a superinfection (infection of HIV-1-infected individuals by another viral strain). We tested if the excluded samples were among potential superinfections previously identified in a systematic screen.32 We also tested our analysis at a stricter threshold of 1.5% and without any filtering based on DNA pairwise distance. Secondly, we assessed the difference between SS and NGS in detection of HIV-DRMs for B and non-B subtypes. Thirdly, we performed several analyses to assess the plausibility of the low-abundance DRMs detected at variant-calling thresholds below 10%: We assessed if PLHIV with HIV-DRMs had had prior ART therapy with or without ARV drugs that select for the detected HIV-DRMs. HIV-DRMs that are detected in ART-naive PLHIV only at lower variant-calling thresholds may either represent transmitted HIV-DRMs that have reverted, de novo appearance of DRMs or sequencing artefacts. We assessed these possibilities by testing if these low-abundance mutations were also detected in previous resistance tests of the same individual.

Statistical analysis

We compared the detection of HIV-DRMs by SS and NGS using Cohen’s kappa, a scalar metric of accuracy, as the parameter to measure the agreement.37 Cohen’s kappa value ranges from −1 to +1. Values closer to 1 indicate high agreement between the measurements, whereas values near to 0 indicate no agreement. We compared the drug resistance levels detected by SS and NGS (ordinal variable) using Light’s kappa (the average of all possible combinations of bivariate kappa). We performed all analyses in R, version 4.1.2 (R Foundation for Statistical Computing, Vienna, Austria).38 We used the ape package in R to handle sequence data and to compute pairwise DNA distance.39 We used the psych package to compute Cohen’s kappa and Light’s kappa.40 We used the boot package to construct bootstrap CI for Light’s kappa.41

Results

Overall, 24 394 SS sequences and 5211 NGS (Illumina) sequences were available from 21 297 PLHIV enrolled in the SHCS as of August 2021. Of these PLHIV, 342 were also enrolled in the ZPHI. There were 816 paired samples (at least one sequence each from SS and NGS available for the pol region of HIV and with a difference in sample collection date for SS and NGS of 7 days or less) from 607 PLHIV. We obtained 709 sample pairs after restricting multiple possible sample pairs within the same 7-day window to a single sample pair. One sample pair was excluded because only the integrase region was covered by the available SS sequence, 62 were excluded because the median depth of available NGS sequences was less than 100 reads per position, and 52 were excluded because the pairwise DNA distance between SS and NGS was ≥4.5% (Figure 1). To understand the impact of these sequences, we performed a sensitivity analysis by varying the pairwise DNA distance threshold (Figure S1) and tested the overlap with previously identified potential superinfections.32 After excluding sequences based on these quality control steps, we obtained 594 sample pairs from 527 PLHIV (Figure 1).

Selection of study population.
Figure 1.

Selection of study population.

The majority (80.5%) of PLHIV included in our study were male (Table 1a). At sample collection, 20.4% of PLHIV were on ART and 76.3% were ART naive. In the 594 sample pairs, 590 (99.3%) SS sequences covered the PR region, 584 (98.3%) SS sequences covered the RT region, and only 19 (3.2%) SS sequences covered the IN region of the HIV genome (Table 1b). The PR region was covered by 584 (98.3%) NGS sequences, 559 (94.1%) NGS sequences covered the RT region, and 559 (94.1%) NGS sequences covered the IN region of the HIV genome. The majority (78.1%) of the sequences were of subtype B (Table 1b).

Table 1.

Characteristics of study population and sequences

(a) Study population
Number of PLHIV, n527
Number of sample pairs, n594
Number of males, n (%)424 (80.5)
Year of birth, median [IQR]1967 [1960–1973]
Year of HIV diagnosis, median [IQR]2004 [1998–2007]
HIV viral load (log10 copies/mL), median [IQR]a4.7 [4.3–5.2]
CD4 cell-counts (cells/µL), median [IQR]a289 (162–450)
On ART, n (%)a121 (20.4)
ART-naive, n (%)a453 (76.3)
(a) Study population
Number of PLHIV, n527
Number of sample pairs, n594
Number of males, n (%)424 (80.5)
Year of birth, median [IQR]1967 [1960–1973]
Year of HIV diagnosis, median [IQR]2004 [1998–2007]
HIV viral load (log10 copies/mL), median [IQR]a4.7 [4.3–5.2]
CD4 cell-counts (cells/µL), median [IQR]a289 (162–450)
On ART, n (%)a121 (20.4)
ART-naive, n (%)a453 (76.3)
(b) Sequences
Sanger sequenceNGS sequence
Total available sequences, n594594
Sample collection year, median [IQR]2006 [2002–2008]2006 [2002–2008]
Year of sequencing, median [IQR]2008 (2007–2012)2016 (2016–2018)
Subtype, n (%)B464 (78.1)465 (78.3)
CRF01_AE34 (5.7)34 (5.7)
C26 (4.4)26 (4.4)
A22 (3.7)23 (3.9)
CRF02_AG20 (3.4)22 (3.7)
Others28 (4.7)24 (4.1)
HIV region covered by sequencing, n (%)PR590 (99.3)584 (98.3)
RT584 (98.3)559 (94.1)
IN19 (3.2)559 (94.1)
Number of amino acids covered by sequencing, median [IQR]PR99 [99–99]99 [98–99]
RT335 [321–335]440 [436–441]
IN288 [288–288]285 [283–288]
(b) Sequences
Sanger sequenceNGS sequence
Total available sequences, n594594
Sample collection year, median [IQR]2006 [2002–2008]2006 [2002–2008]
Year of sequencing, median [IQR]2008 (2007–2012)2016 (2016–2018)
Subtype, n (%)B464 (78.1)465 (78.3)
CRF01_AE34 (5.7)34 (5.7)
C26 (4.4)26 (4.4)
A22 (3.7)23 (3.9)
CRF02_AG20 (3.4)22 (3.7)
Others28 (4.7)24 (4.1)
HIV region covered by sequencing, n (%)PR590 (99.3)584 (98.3)
RT584 (98.3)559 (94.1)
IN19 (3.2)559 (94.1)
Number of amino acids covered by sequencing, median [IQR]PR99 [99–99]99 [98–99]
RT335 [321–335]440 [436–441]
IN288 [288–288]285 [283–288]

At sample collection

Table 1.

Characteristics of study population and sequences

(a) Study population
Number of PLHIV, n527
Number of sample pairs, n594
Number of males, n (%)424 (80.5)
Year of birth, median [IQR]1967 [1960–1973]
Year of HIV diagnosis, median [IQR]2004 [1998–2007]
HIV viral load (log10 copies/mL), median [IQR]a4.7 [4.3–5.2]
CD4 cell-counts (cells/µL), median [IQR]a289 (162–450)
On ART, n (%)a121 (20.4)
ART-naive, n (%)a453 (76.3)
(a) Study population
Number of PLHIV, n527
Number of sample pairs, n594
Number of males, n (%)424 (80.5)
Year of birth, median [IQR]1967 [1960–1973]
Year of HIV diagnosis, median [IQR]2004 [1998–2007]
HIV viral load (log10 copies/mL), median [IQR]a4.7 [4.3–5.2]
CD4 cell-counts (cells/µL), median [IQR]a289 (162–450)
On ART, n (%)a121 (20.4)
ART-naive, n (%)a453 (76.3)
(b) Sequences
Sanger sequenceNGS sequence
Total available sequences, n594594
Sample collection year, median [IQR]2006 [2002–2008]2006 [2002–2008]
Year of sequencing, median [IQR]2008 (2007–2012)2016 (2016–2018)
Subtype, n (%)B464 (78.1)465 (78.3)
CRF01_AE34 (5.7)34 (5.7)
C26 (4.4)26 (4.4)
A22 (3.7)23 (3.9)
CRF02_AG20 (3.4)22 (3.7)
Others28 (4.7)24 (4.1)
HIV region covered by sequencing, n (%)PR590 (99.3)584 (98.3)
RT584 (98.3)559 (94.1)
IN19 (3.2)559 (94.1)
Number of amino acids covered by sequencing, median [IQR]PR99 [99–99]99 [98–99]
RT335 [321–335]440 [436–441]
IN288 [288–288]285 [283–288]
(b) Sequences
Sanger sequenceNGS sequence
Total available sequences, n594594
Sample collection year, median [IQR]2006 [2002–2008]2006 [2002–2008]
Year of sequencing, median [IQR]2008 (2007–2012)2016 (2016–2018)
Subtype, n (%)B464 (78.1)465 (78.3)
CRF01_AE34 (5.7)34 (5.7)
C26 (4.4)26 (4.4)
A22 (3.7)23 (3.9)
CRF02_AG20 (3.4)22 (3.7)
Others28 (4.7)24 (4.1)
HIV region covered by sequencing, n (%)PR590 (99.3)584 (98.3)
RT584 (98.3)559 (94.1)
IN19 (3.2)559 (94.1)
Number of amino acids covered by sequencing, median [IQR]PR99 [99–99]99 [98–99]
RT335 [321–335]440 [436–441]
IN288 [288–288]285 [283–288]

At sample collection

For each of the 594 sample pairs, we assessed the occurrence of 53 DRMs in the PI and RT region that confer at least low-level resistance to a PI, NRTI or NNRTI drug by themselves (Table S2). Of the 31 482 positions screened for potential occurrences of HIV-DRMs, 28 108 were covered by both NGS and SS. As expected, the number of positions with HIV-DRMs detected only by NGS decreased with an increase in variant-calling threshold (from 427 positions at 1% to 17 positions at 30% threshold) (Figure 2). Agreement (Cohen’s kappa) between SS and NGS in detecting HIV-DRM ranged from 0.61 (95% CI: 0.58 to 0.64) at 1% variant-calling threshold to 0.89 (95% CI: 0.87 to 0.91) at 30% threshold (Figure 3). Overall, a good agreement (Cohen’s kappa >0.80) was observed for PI, NRTI and NNRTI HIV-DRMs for variant-calling thresholds above 5%. The variant-calling threshold yielding the strongest agreement and the level of the strongest agreement varied across mutations (Figure S2).

HIV-DRMs detected by SS and NGS at different thresholds. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.
Figure 2.

HIV-DRMs detected by SS and NGS at different thresholds. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Agreement between SS and NGS in detection of HIV-DRMs. Agreement is defined by Cohen’s kappa. A Cohen’s kappa value closer to 1 indicates high agreement. The vertical bars indicate the CI of Cohen’s kappas at 1% variant-calling threshold. An HIV-DRM is any mutation of HIV genes conferring at least low-level resistance to PI, NRTI or NNRTI drugs by itself. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.
Figure 3.

Agreement between SS and NGS in detection of HIV-DRMs. Agreement is defined by Cohen’s kappa. A Cohen’s kappa value closer to 1 indicates high agreement. The vertical bars indicate the CI of Cohen’s kappas at 1% variant-calling threshold. An HIV-DRM is any mutation of HIV genes conferring at least low-level resistance to PI, NRTI or NNRTI drugs by itself. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

The number of HIV-DRMs detected by NGS increased with decrease in variant-calling threshold, especially at 1% (Figure 4, Table S4a). M46(ILV), T215(ACDEFILNSVY) and E138(AGKQR) were the most commonly observed PI, NRTI and NNRTI-DRMs, respectively. We observed 28/417 (6.7%) low-abundance HIV-DRMs at 10% threshold but not at 25% threshold. We additionally found 31/448 (6.9%), 27/475 (5.7%), 44/519 (8.4%) and 293/812 (36.1%) low-abundance HIV-DRMs at 5%–10%, 3%–5%, 2%–3% and 1%–2% threshold, respectively (Table S4b).

Number of HIV-DRMs detected by NGS for different variant-calling thresholds. The seven most prevalent HIV-DRMs for each drug class are shown. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.
Figure 4.

Number of HIV-DRMs detected by NGS for different variant-calling thresholds. The seven most prevalent HIV-DRMs for each drug class are shown. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

We further assessed the resistance levels of individual ARV drugs for each of the 594 sample pairs. We found 166 (27.9%) of 578 SS sequences with at least low-level resistance to at least one ARV drug (Table 2). As expected, the number of NGS sequences for which we detected at least low-level resistance decreased with an increase in variant-calling threshold [from 303 (53.3%) at 1% threshold to 150 (26.4%) at 30% threshold]. Agreement (Light’s kappa) between SS and NGS in detecting resistance levels of a sample ranged from 0.48 (95% CI: 0.43 to 0.52) at 1% variant-calling threshold to 0.92 (95% CI: 0.89 to 0.94) at 30% threshold (Figure 5). Overall, a good agreement (Light’s kappa >0.80) was observed for PI, NRTI and NNRTI resistance level for variant-calling threshold at or above 5% (Figure 5). The variant-calling threshold yielding the strongest agreement and the level of the strongest agreement varied across ARV drugs (Figure S3).

Agreement between SS and NGS in detecting resistance level to ARV drugs. Agreement is defined by Light’s kappa. A Light’s kappa value closer to 1 indicates high agreement. The vertical bars indicate the CI of Light’s kappas at 1% variant-calling threshold. ARV denotes any PI, NRTI or NNRTI drug. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.
Figure 5.

Agreement between SS and NGS in detecting resistance level to ARV drugs. Agreement is defined by Light’s kappa. A Light’s kappa value closer to 1 indicates high agreement. The vertical bars indicate the CI of Light’s kappas at 1% variant-calling threshold. ARV denotes any PI, NRTI or NNRTI drug. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region. This figure appears in colour in the online version of JAC and in black and white in the print version of JAC.

Table 2.

Samples with at least low-level resistance observed in SS and NGS at different thresholds

Drug classSS, n (%)NGS at 1%, n (%)NGS at 2%, n (%)NGS at 3%, n (%)NGS at 5%, n (%)NGS at 10%, n (%)NGS at 15%, n (%)NGS at 20%, n (%)NGS at 25%, n (%)NGS at 30%, n (%)
PI59 (10)122 (21.9)71 (12.7)63 (11.3)58 (10.4)54 (9.7)52 (9.3)51 (9.1)51 (9.1)49 (8.8)
NRTI90 (15.4)191 (33.7)129 (22.8)116 (20.5)109 (19.2)96 (16.9)91 (16)88 (15.5)87 (15.3)86 (15.2)
NNRTI62 (10.6)163 (28.7)91 (16)78 (13.8)71 (12.5)66 (11.6)62 (10.9)60 (10.6)60 (10.6)56 (9.9)
INSTI0 (0)174 (30)115 (19.8)98 (16.9)73 (12.6)53 (9.1)32 (5.5)24 (4.1)15 (2.6)10 (1.7)
Any ARVa166 (27.9)303 (53.3)219 (38.6)195 (34.3)182 (32)166 (29.2)159 (28)155 (27.3)154 (27.1)150 (26.4)
Drug classSS, n (%)NGS at 1%, n (%)NGS at 2%, n (%)NGS at 3%, n (%)NGS at 5%, n (%)NGS at 10%, n (%)NGS at 15%, n (%)NGS at 20%, n (%)NGS at 25%, n (%)NGS at 30%, n (%)
PI59 (10)122 (21.9)71 (12.7)63 (11.3)58 (10.4)54 (9.7)52 (9.3)51 (9.1)51 (9.1)49 (8.8)
NRTI90 (15.4)191 (33.7)129 (22.8)116 (20.5)109 (19.2)96 (16.9)91 (16)88 (15.5)87 (15.3)86 (15.2)
NNRTI62 (10.6)163 (28.7)91 (16)78 (13.8)71 (12.5)66 (11.6)62 (10.9)60 (10.6)60 (10.6)56 (9.9)
INSTI0 (0)174 (30)115 (19.8)98 (16.9)73 (12.6)53 (9.1)32 (5.5)24 (4.1)15 (2.6)10 (1.7)
Any ARVa166 (27.9)303 (53.3)219 (38.6)195 (34.3)182 (32)166 (29.2)159 (28)155 (27.3)154 (27.1)150 (26.4)

INSTI, integrase inhibitors.

ARV denotes any PI, NRTI or NNRTI drug. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region.

Table 2.

Samples with at least low-level resistance observed in SS and NGS at different thresholds

Drug classSS, n (%)NGS at 1%, n (%)NGS at 2%, n (%)NGS at 3%, n (%)NGS at 5%, n (%)NGS at 10%, n (%)NGS at 15%, n (%)NGS at 20%, n (%)NGS at 25%, n (%)NGS at 30%, n (%)
PI59 (10)122 (21.9)71 (12.7)63 (11.3)58 (10.4)54 (9.7)52 (9.3)51 (9.1)51 (9.1)49 (8.8)
NRTI90 (15.4)191 (33.7)129 (22.8)116 (20.5)109 (19.2)96 (16.9)91 (16)88 (15.5)87 (15.3)86 (15.2)
NNRTI62 (10.6)163 (28.7)91 (16)78 (13.8)71 (12.5)66 (11.6)62 (10.9)60 (10.6)60 (10.6)56 (9.9)
INSTI0 (0)174 (30)115 (19.8)98 (16.9)73 (12.6)53 (9.1)32 (5.5)24 (4.1)15 (2.6)10 (1.7)
Any ARVa166 (27.9)303 (53.3)219 (38.6)195 (34.3)182 (32)166 (29.2)159 (28)155 (27.3)154 (27.1)150 (26.4)
Drug classSS, n (%)NGS at 1%, n (%)NGS at 2%, n (%)NGS at 3%, n (%)NGS at 5%, n (%)NGS at 10%, n (%)NGS at 15%, n (%)NGS at 20%, n (%)NGS at 25%, n (%)NGS at 30%, n (%)
PI59 (10)122 (21.9)71 (12.7)63 (11.3)58 (10.4)54 (9.7)52 (9.3)51 (9.1)51 (9.1)49 (8.8)
NRTI90 (15.4)191 (33.7)129 (22.8)116 (20.5)109 (19.2)96 (16.9)91 (16)88 (15.5)87 (15.3)86 (15.2)
NNRTI62 (10.6)163 (28.7)91 (16)78 (13.8)71 (12.5)66 (11.6)62 (10.9)60 (10.6)60 (10.6)56 (9.9)
INSTI0 (0)174 (30)115 (19.8)98 (16.9)73 (12.6)53 (9.1)32 (5.5)24 (4.1)15 (2.6)10 (1.7)
Any ARVa166 (27.9)303 (53.3)219 (38.6)195 (34.3)182 (32)166 (29.2)159 (28)155 (27.3)154 (27.1)150 (26.4)

INSTI, integrase inhibitors.

ARV denotes any PI, NRTI or NNRTI drug. INSTI-DRM were excluded because only 3.2% of SS sequences covered the integrase region.

We excluded 52 samples during quality control because the pairwise DNA distance between SS and NGS was ≥4.5%. Of these 52 samples, 27 (51.9%) were previously identified as potential HIV superinfection based on multiple available NGS sequences.32 Overall we observed similar results in sensitivity analyses using either a stricter threshold of ≥1.5%, or without excluding any sequences based on the distance between the NGS and SS sequences (Figure S1): as expected, the agreement between SS and NGS in detecting HIV-DRMs decreased in the absence of exclusion based on pairwise DNA distance (Figure S1), whereas it slightly increased when using a 1.5% instead of a 4.5% pairwise distance threshold.

We assessed the agreement between SS and NGS in detecting HIV-DRMs for B and non-B subtypes. At all thresholds the agreement was lower for non-B subtypes than for subtype B (Figure S4), especially for a variant-calling threshold of 1% [Cohen’s kappa 0.70 (95% CI: 0.67–0.74) for subtype B versus 0.24 (95% CI: 0.21–0.28) for non-B subtypes]. However, the difference in agreement reduced with an increase in the threshold [at 30% variant-calling threshold: Cohen’s kappa 0.89 (95% CI: 0.87–0.91) for subtype B versus 0.87 (95% CI: 0.77–0.96) for non-B subtypes].

We further assessed the plausibility of the low-abundance DRMs detected at lower (≤10%) variant-calling thresholds. Of the 389 HIV-DRMs detected by NGS at a 25% threshold, 155 (39.8%) were detected in ART-naive PLHIV. Of the other 234 PLHIV, 48 (20.5%) were detected in PLHIV who received prior ARV drugs that do not select for the detected HIV-DRMs, and 186 (79.5%) were detected in PLHIV who received prior ARV drugs that select for the detected HIV-DRMs (Table S4). At a lower (≤10%) variant-calling threshold, the proportion of HIV-DRMs detected in ART-naive PLHIV increased and the proportion of HIV-DRMs detected in those who received a prior ARV drug that selects for detected HIV-DRMs decreased (Table S4). We further assessed the plausibility of low-abundance HIV-DRMs by checking whether these mutations were also detected in a previous sequence available for the same individual (and hence representing a reversion). The proportion of low-abundance HIV-DRMs detected also detected at a previously available sequence decreased with decrease in variant-calling thresholds [1/3 (33.3%) at 10%–25% to 1/60 (1.7%) at 10%–25% threshold] (Table S5), suggesting that low-abundance HIV-DRMs detected in ART-naive PLHIV, especially at very low thresholds (<3%), are less likely to be transmitted HIV-DRMs.

Discussion

In this study, we compared the agreement between SS and NGS in detecting HIV-DRMs and different resistance levels to ARV drugs. We further assessed the presence of low-abundance HIV-DRMs and their dependency on the variant-calling thresholds. Overall, we found a good agreement (Cohen’s kappa and Light’s kappa >0.8) for DRMs conferring resistance to PIs, NRTIs and NNRTIs with variant-calling thresholds above 5%.

SS typically detects HIV-DRMs present in at least 15%–25% of variants of circulating viruses in the plasma,10 whereas NGS can detect HIV-DRMs present even when present in lower proportions of circulating viruses. The lower the variant-calling threshold, the higher the number of low-abundance mutations detected.9 However, variant-calling thresholds <1% are often not considered because many of the variants at this threshold may be due to errors introduced during PCR amplification and sequencing.42 Some previously established pipelines therefore even suggest using a threshold of 5% to minimize variants due to sequencing error.16,43 However, there is no clear evidence for thresholds between 1% and 5%. Results from a cross-sectional study in PLHIV in China showed a sharp increase in low-abundance HIV-DRMs when the variant-calling threshold was below 5%.44 Our results were in line with these previous studies.

We performed additional analysis to assess the validity of the low-abundance HIV-DRMs detected at lower (≤10%) variant-calling thresholds. Higher proportions of low-abundance HIV-DRMs were found in ART-naive PLHIV, particularly at 1%–2% and 2%–3% thresholds compared with 3%–5%, 5%–10% and 10%–25%, suggesting that these mutations are transmitted HIV-DRMs or due to sequencing error. We further assessed these possibilities by testing if the low-abundance mutations were detected at a higher threshold at a previous timepoint. However, such low-abundance HIV-DRMs were overrepresented in ART-naive participants and were in most cases not detected in previously sampled sequences for thresholds <3%. These findings suggest that these low-abundance HIV-DRMs are less likely to be transmitted HIV-DRMs that may have reverted, but rather artifacts due to sequencing errors or de novo appearance, and hence these low-abundance HIV-DRMs should not be overinterpreted in clinical practice. We found a small yet non-negligible fraction of low-abundance HIV-DRMs detected by NGS for thresholds between 3% and 10% whose clinical relevance warrants further investigation. Several studies have shown that such low-abundance mutations are associated with virological failure in PLHIV on ART, especially for NNRTIs.45–48

We excluded sample pairs with a large pairwise DNA distance threshold to ensure high quality. Fifty-one percent of the excluded samples were potential superinfections (previously described by Chaudron et al.32). These were overrepresented in our initial study population because one of the projects contributing to the NGS database explicitly aimed at sequencing such potential superinfections.32 These potential superinfections were successfully filtered out using our quality control because the exclusion criteria were slightly more encompassing than the criteria used by Chaudron et al.32 to define superinfections.

Our study had several limitations. Firstly, in our study there were only a small number of sequence pairs for which the SS sequences contained the integrase region. Hence, we had to exclude INSTI-DRMs in analyses comparing the overall agreement between SS and NGS. Secondly, because most of the sequences of our study were of subtype B, we did not have statistical power to assess the difference in agreement between SS and NGS in different HIV subtypes in detail. Despite these limitations, this study is the largest to systematically assess the agreement between SS and NGS in detecting HIV-DRMs and resistance levels to ARV drugs for the sequences collected in a clinical setting. Our results show a high agreement between SS and NGS, with NGS further detecting a substantial number of low-abundance mutations.

Conclusions

We observed, for different variant-calling thresholds above 5%, a good concordance (Cohen’s kappa and Light’s kappa >0.8) between SS and NGS in detecting HIV-DRMs as well as for the resulting resistance levels to ARV drugs. We observed a substantial number of low-abundance HIV-DRMs detected only by NGS at lower variant-calling thresholds. Our findings suggest that a substantial fraction of the low-abundance HIV-DRMs detected at thresholds <3% may represent sequencing errors and hence should not be overinterpreted in clinical practice.

Acknowledgements

We thank the patients for participating in the SHCS, the study nurses, physicians, data managers and the administrative assistants. The members of the SHCS are: A. Anagnostopoulos, M. Battegay, E. Bernasconi, J. Böni, D.L. Braun, H.C. Bucher, A. Calmy, M. Cavassini, A. Ciuffi, G. Dollenmaier, M. Egger, L. Elzi, J. Fehr, J. Fellay, H. Furrer, C.A. Fux, H. Günthard (President of the SHCS), D. Haerry (deputy of ‘Positive Council’), B. Hasse, H.H. Hirsch, M. Hoffmann, I. Hösli, M. Huber, C.R. Kahlert (Chairman of the Mother and Child Substudy), L. Kaiser, O. Keiser, T. Klimkait, R.D. Kouyos, H. Kovari, K. Kusejko (Head of Data Centre), B. Ledergerber, G. Martinetti, B. Martinez de Tejada, C. Marzolini, K.J. Metzner, N. Müller, D. Nicca, P. Paioni, G. Pantaleo, M. Perreau, A. Rauch (Chairman of the Scientific Board), C. Rudin, P. Schmid, R. Speck, M. Stöckle (Chairman of the Clinical and Laboratory Committee), P. Tarr, A. Trkola, P. Vernazza, G. Wandeler, R. Weber and S. Yerly.

Funding

This work was supported by the Swiss National Science Foundation (grant numbers 33CS30_177499 to H.F.G. in the framework of the Swiss HIV Cohort Study [SHCS]; 324730B_179571 and 310030_141067 to H.F.G.; and 324730_207957 and BSSGI0_155851 to R.D.K.); the Yvonne-Jacob Foundation (to H.F.G.); the University of Zurich Clinical Research Priority Program for Viral Infectious Disease, the Zurich Primary HIV Infection Cohort Study (to H.F.G.); and an unrestricted research grant from Gilead Sciences (to the SHCS Research Foundation).

Transparency declarations

K.J.M. has received travel grants and honoraria from Gilead Sciences, Roche Diagnostics, GlaxoSmithKline, Merck Sharp & Dohme (MSD), Bristol-Myers Squibb, ViiV and Abbott; and the University of Zurich received research grants from Gilead Science, Novartis, Roche and MSD for studies in which K.J.M. serves as principal investigator, and advisory board honoraria from Gilead Sciences and ViiV. M.C.’s institution received research grants from Gilead, MSD and Viiv. A.R. reports support to his institution for advisory boards and/or travel grants from MSD, Gilead Sciences, Pfizer and Abbvie, and an investigator-initiated trial (IIT) grant from Gilead Sciences. All remuneration went to his home institution and not to A.R. personally, and all remuneration was provided outside the submitted work. R.D.K. has received grants from the Swiss National Science Foundation (for this work) and Gilead Sciences (outside the submitted work). H.F.G has received unrestricted research grants from the Swiss National Science Foundation, NIH, Yvonne Jacob Foundation and Gilead Sciences; fees for data and safety monitoring board, consultancy or advisory board membership from Merck, Gilead, ViiV, Johnson and Johnson, Janssen, and Novartis. All other coauthors declare no conflict of interests.

Author contributions

S.B. and R.D.K. developed the study protocol and drafted the manuscript. Remaining authors contributed materials and methdos and critically reviewed the study protocol and the manuscript. S.B. performed the statistical analyses and was the principal investigator. All authors contributed to the design of the study and approved the final version of the manuscript.

Supplementary data

Figures S1 to S4 and Tables S1 to S5 are available as Supplementary data at JAC online.

References

1

Gueler
A
,
Moser
A
,
Calmy
A
et al.
Life expectancy in HIV-positive persons in Switzerland: matched comparison with general population
.
AIDS
2017
;
31
:
427
36
. https://doi.org/10.1097/QAD.0000000000001335

2

Cohen
MS
,
Chen
YQ
,
McCauley
M
et al.
Antiretroviral therapy for the prevention of HIV-1 transmission
.
N Engl J Med
2016
;
375
:
830
9
. https://doi.org/10.1056/NEJMoa1600693

3

Wittkop
L
,
Günthard
HF
,
de Wolf
F
et al.
Effect of transmitted drug resistance on virological and immunological response to initial combination antiretroviral therapy for HIV (EuroCoord-CHAIN joint project): a European multicohort study
.
Lancet Infect Dis
2011
;
11
:
363
71
. https://doi.org/10.1016/S1473-3099(11)70032-9

4

Vandamme
A-M
,
Camacho
RJ
,
Ceccherini-Silberstein
F
et al.
European recommendations for the clinical use of HIV drug resistance testing: 2011 update
.
AIDS Rev
2011
;
13
:
77
108
.

5

Kitahata
MM
,
Gange
SJ
,
Abraham
AG
et al.
Effect of early versus deferred antiretroviral therapy for HIV on survival
.
N Engl J Med
2009
;
360
:
1815
26
. https://doi.org/10.1056/NEJMoa0807252

6

Weinstein
MC
,
Goldie
SJ
,
Losina
E
et al.
Use of genotypic resistance testing to guide HIV therapy: clinical impact and cost-effectiveness
.
Ann Intern Med
2001
;
134
:
440
. https://doi.org/10.7326/0003-4819-134-6-200103200-00008

7

Günthard
HF
,
Calvez
V
,
Paredes
R
et al.
Human immunodeficiency virus drug resistance: 2018 recommendations of the International Antiviral Society—USA panel
.
Clin Infect Dis
2019
;
68
:
177
87
. https://doi.org/10.1093/cid/ciy463

8

Larder
BA
,
Kohli
A
,
Kellam
P
et al.
Quantitative detection of HIV-1 drug resistance mutations by automated DNA sequencing
.
Nature
1993
;
365
:
671
3
. https://doi.org/10.1038/365671a0

9

Tzou
PL
,
Ariyaratne
P
,
Varghese
V
et al.
Comparison of an in vitro diagnostic next-generation sequencing assay with Sanger sequencing for HIV-1 genotypic resistance testing
.
J Clin Microbiol
2018
;
56
: e00105-18. https://doi.org/10.1128/JCM.00105-18

10

Schuurman
R
,
Demeter
L
,
Reichelderfer
P
et al.
Worldwide evaluation of DNA sequencing approaches for identification of drug resistance mutations in the human immunodeficiency virus type 1 reverse transcriptase
.
J Clin Microbiol
1999
;
37
:
2291
6
. https://doi.org/10.1128/JCM.37.7.2291-2296.1999

11

Beerenwinkel
N
,
Günthard
H
,
Roth
V
,
Metzner
K
.
Challenges and opportunities in estimating viral genetic diversity from next-generation sequencing data
.
Front Microbiol
2012
;
3
: 329. https://doi.org/10.3389/fmicb.2012.00329

12

Lapointe
HR
,
Dong
W
,
Lee
GQ
et al.
HIV Drug resistance testing by high-multiplex “wide” sequencing on the MiSeq instrument
.
Antimicrob Agents Chemother
2015
;
59
:
6824
33
. https://doi.org/10.1128/AAC.01490-15

13

Metzner
KJ
.
Technologies for HIV-1 drug resistance testing: inventory and needs
.
Curr Opin HIV AIDS
2022
;
17
:
222
8
. https://doi.org/10.1097/COH.0000000000000737

14

Alidjinou
EK
,
Deldalle
J
,
Hallaert
C
et al.
RNA and DNA Sanger sequencing versus next-generation sequencing for HIV-1 drug resistance testing in treatment-naive patients
.
J Antimicrob Chemother
2017
;
72
:
2823
30
. https://doi.org/10.1093/jac/dkx232

15

Fokam
J
,
Bellocchi
MC
,
Armenia
D
et al.
Next-generation sequencing provides an added value in determining drug resistance and viral tropism in Cameroonian HIV-1 vertically infected children
.
Medicine (Baltimore)
2018
;
97
: e0176. https://doi.org/10.1097/MD.0000000000010176

16

Trabaud
M-A
,
Icard
V
,
Ramière
C
et al.
Comparison of HIV-1 drug-resistance genotyping by ultra-deep sequencing and Sanger sequencing using clinical samples
.
J Med Virol
2017
;
89
:
1912
–1
9
. https://doi.org/10.1002/jmv.24872

17

Moscona
R
,
Ram
D
,
Wax
M
et al.
Comparison between next-generation and Sanger-based sequencing for the detection of transmitted drug-resistance mutations among recently infected HIV-1 patients in Israel, 2000–2014
.
J Int AIDS Soc
2017
;
20
:
21846
. https://doi.org/10.7448/IAS.20.1.21846

18

Tekin
D
,
Gokengin
D
,
Onay
H
et al.
Investigation of drug resistance against protease, reverse transcriptase, and integrase inhibitors by next-generation sequencing in HIV-positive patients
.
J Med Virol
2021
;
93
:
3627
33
. https://doi.org/10.1002/jmv.26582

19

Telele
NF
,
Kalu
AW
,
Gebre-Selassie
S
et al.
Pretreatment drug resistance in a large countrywide Ethiopian HIV-1C cohort: a comparison of Sanger and high-throughput sequencing
.
Sci Rep
2018
;
8
:
7556
. https://doi.org/10.1038/s41598-018-25888-6

20

Kireev
DE
,
Lopatukhin
AE
,
Murzakova
AV
et al.
Evaluating the accuracy and sensitivity of detecting minority HIV-1 populations by Illumina next-generation sequencing
.
J Virol Methods
2018
;
261
:
40
5
. https://doi.org/10.1016/j.jviromet.2018.08.001

21

Parkin
NT
,
Avila-Rios
S
,
Bibby
DF
et al.
Multi-laboratory comparison of next-generation to Sanger-based sequencing for HIV-1 drug resistance genotyping
.
Viruses
2020
;
12
:
694
. https://doi.org/10.3390/v12070694

22

Taylor
T
,
Lee
ER
,
Nykoluk
M
et al.
A MiSeq-HyDRA platform for enhanced HIV drug resistance genotyping and surveillance
.
Sci Rep
2019
;
9
:
8970
. https://doi.org/10.1038/s41598-019-45328-3

23

Scherrer
AU
,
Traytel
A
,
Braun
DL
et al.
Cohort profile update: the Swiss HIV Cohort Study (SHCS)
.
Int J Epidemiol
2022
;
51
:
33
34j
. https://doi.org/10.1093/ije/dyab141

24

Rieder
P
,
Joos
B
,
Scherrer
AU
et al.
Characterization of human immunodeficiency virus type 1 (HIV-1) diversity and tropism in 145 patients with primary HIV-1 infection
.
Clin Infect Dis
2011
;
53
:
1271
9
. https://doi.org/10.1093/cid/cir725

25

Schmutz
S
,
Huber
M
,
Zaheri
M
,
Zagordi
O
.
SmaltAlign
.
2021
. https://github.com/medvir/SmaltAlign

26

Kletenkov
K
,
Hoffmann
D
,
Böni
J
et al.
Role of Gag mutations in PI resistance in the Swiss HIV cohort study: bystanders or contributors?
J Antimicrob Chemother
2017
;
72
:
866
75
. https://doi.org/10.1093/jac/dkw493

27

Yerly
S
,
von Wyl
V
,
Ledergerber
B
et al.
Transmission of HIV-1 drug resistance in Switzerland: a 10-year molecular epidemiology survey
.
AIDS
2007
;
21
:
2223
9
. https://doi.org/10.1097/QAD.0b013e3282f0b685

28

Giallonardo
FD
,
Töpfer
A
,
Rey
M
et al.
Full-length haplotype reconstruction to infer the structure of heterogeneous virus populations
.
Nucleic Acids Res
2014
;
42
:
e115
. https://doi.org/10.1093/nar/gku537

29

Gall
A
,
Ferns
B
,
Morris
C
et al.
Universal amplification, next-generation sequencing, and assembly of HIV-1 genomes
.
J Clin Microbiol
2012
;
50
:
3838
44
. https://doi.org/10.1128/JCM.01516-12

30

Carlisle
LA
,
Turk
T
,
Kusejko
K
et al.
Viral diversity based on next-generation sequencing of HIV-1 provides precise estimates of infection recency and time since infection
.
J Infect Dis
2019
;
220
:
254
65
. https://doi.org/10.1093/infdis/jiz094

31

Wan
C
,
Bachmann
N
,
Mitov
V
et al.
Heritability of the HIV-1 reservoir size and decay under long-term suppressive ART
.
Nat Commun
2020
;
11
:
5542
. https://doi.org/10.1038/s41467-020-19198-7

32

Chaudron
SE
,
Leemann
C
,
Kusejko
K
et al.
A systematic molecular epidemiology screen reveals numerous HIV-1 superinfections in the Swiss HIV cohort study
.
J Infect Dis
2022
;
226
:
1256
66
. https://doi.org/10.1093/infdis/jiac166

33

Blanquart
F
,
Wymant
C
,
Cornelissen
M
et al.
Viral genetic variation accounts for a third of variability in HIV-1 set-point viral load in Europe
.
PLoS Biol
2017
;
15
:
e2001855
. https://doi.org/10.1371/journal.pbio.2001855

34

Cornelissen
M
,
Gall
A
,
Vink
M
et al.
From clinical sample to complete genome: comparing methods for the extraction of HIV-1 RNA for high-throughput deep sequencing
.
Virus Res
2017
;
239
:
10
16
. https://doi.org/10.1016/j.virusres.2016.08.004

35

Rindler
AE
,
Kuster
H
,
Neumann
K
et al.
A novel high throughput, parallel infection assay for determining the replication capacities of 346 primary HIV-1 isolates of the Zurich primary HIV-1 infection study in primary cells
.
Viruses
2021
;
13
:
404
. https://doi.org/10.3390/v13030404

36

Liu
TF
,
Shafer
RW
.
Web resources for HIV type 1 genotypic-resistance test interpretation
.
Clin Infect Dis
2006
;
42
:
1608
18
. https://doi.org/10.1086/503914

37

Cohen
J
.
A coefficient of agreement for nominal scales
.
Educ Psychol Meas
1960
;
20
:
37
46
. https://doi.org/10.1177/001316446002000104

38

R Core Team
.
R: A Language and Environment for Statistical Computing
.
R Foundation for Statistical Computing
,
2020
. https://www.R-project.org/

39

Paradis
E
,
Schliep
K
.
Ape 5.0: an environment for modern phylogenetics and evolutionary analyses in R
.
Bioinformatics
2019
;
35
:
526
8
. https://doi.org/10.1093/bioinformatics/bty633

41

Canty
A
,
Ripley
B
.
boot: Bootstrap R (S-Plus) functions. R package version 1.3-28
;
Cambridge University Press; 2021
.

42

Thys
K
,
Verhasselt
P
,
Reumers
J
et al.
Performance assessment of the Illumina massively parallel sequencing platform for deep sequencing analysis of viral minority variants
.
J Virol Methods
2015
;
221
:
29
38
. https://doi.org/10.1016/j.jviromet.2015.04.022

43

Huber
M
,
Metzner
KJ
,
Geissberger
FD
et al.
Minvar: a rapid and versatile tool for HIV-1 drug resistance genotyping by deep sequencing
.
J Virol Methods
2017
;
240
:
7
13
. https://doi.org/10.1016/j.jviromet.2016.11.008

44

Li
M
,
Liang
S
,
Zhou
C
et al.
HIV drug resistance mutations detection by next-generation sequencing during antiretroviral therapy interruption in China
.
Pathogens
2021
;
10
:
264
. https://doi.org/10.3390/pathogens10030264

45

Paredes
R
,
Lalama
CM
,
Ribaudo
HJ
et al.
Pre-existing minority drug-resistant HIV-1 variants, adherence, and risk of antiretroviral treatment failure
.
J Infect Dis
2010
;
201
:
662
71
. https://doi.org/10.1086/650543

46

Cozzi-Lepri
A
,
Noguera-Julian
M
,
Di Giallonardo
F
et al.
Low-frequency drug-resistant HIV-1 and risk of virological failure to first-line NNRTI-based ART: a multicohort European case–control study using centralized ultrasensitive 454 pyrosequencing
.
J Antimicrob Chemother
2015
;
70
:
930
40
. https://doi.org/10.1093/jac/dku426

47

Kyeyune
F
,
Gibson
RM
,
Nankya
I
et al.
Low-frequency drug resistance in HIV-infected Ugandans on antiretroviral treatment is associated with regimen failure
.
Antimicrob Agents Chemother
2016
;
60
:
3380
97
. https://doi.org/10.1128/AAC.00038-16

48

Mbunkah
HA
,
Bertagnolio
S
,
Hamers
RL
et al.
Low-abundance drug-resistant HIV-1 variants in antiretroviral drug-naive individuals: a systematic review of detection methods, prevalence, and clinical impact
.
J Infect Dis
2020
;
221
:
1584
97
. https://doi.org/10.1093/infdis/jiz650

This article is published and distributed under the terms of the Oxford University Press, Standard Journals Publication Model (https://academic.oup.com/journals/pages/open_access/funder_policies/chorus/standard_publication_model)

Supplementary data