How Generalizable Are Randomized Controlled Trials (RCTs) in Staphylococcus aureus Bacteremia? A Description of the Mortality Gap Between RCTs and Observational Studies 

Abstract

In Staphylococcus aureus bacteremia, mortality rates in randomized controlled trials (RCTs) are consistently lower than observational studies. Stringent eligibility criteria and omission of early deaths in RCTs contribute to this mortality gap. Clinicians should acknowledge the possibility of a lower treatment effect when applying RCT results to bedside care.

Staphylococcus aureus, bacteremia, mortality, trials

To evaluate the effect of interventions on mortality in Staphylococcus aureus bacteremia (SAB), there have been increasing efforts to conduct pragmatic randomized controlled trials (RCTs). The results from these RCTs are intended to inform SAB management at the bedside. Yet to be generalizable to clinical practice, patients in pragmatic RCTs should be similar to patients who are going to be treated outside of trials in terms of baseline characteristics and prognosis. Unfortunately, this may not be the case in RCTs for several reasons. For example, external validity is reduced when eligibility criteria are stringent or if a delay due to consent and enrollment processes omits the sickest patients who died early. This will be amplified in trials where enrollment occurs later into the treatment course, such as trials evaluating stepdown to oral antibiotics or antibiotic duration.

By contrast, observational studies on SAB are more likely to represent the general SAB patient population with respect to mortality, because most observational studies recruit consecutive patients with minimal exclusion criteria and typically capture all patients from the time of blood culture collection—including those who die early.

We recently conducted a systematic review and meta-analysis on mortality in SAB [1]. In this secondary analysis, we compared the all-cause mortality rates observed in RCTs versus observational studies. We hypothesized that there is a substantial gap in the observed mortality between observational studies and RCTs. This would be important when translating trial results to patient care at the bedside and when designing future RCTs.

METHODS

The systematic review protocol was registered (PROSPERO CRD42021253891) and described elsewhere [1]. To summarize, a literature search was performed using MEDLINE, Embase, and the Cochrane Database of Systematic Reviews for dates between 1 January 1991 and 7 May 2021 using MeSH terms to capture S. aureus, bacteremia, and mortality. Studies (RCTs and observational studies) that included patients with SAB based on positive blood culture(s) and reported numbers for all-cause mortality were included. Studies that included bacteremia due to bacteria other than S. aureus were excluded. Two independent reviewers extracted the data in duplicate.

In this secondary analysis, RCTs were compared to observational studies in 2 ways. First, RCTs were compared to all observational studies. In our original systematic review [1], a meta-regression model showed that the decade and geographic continent that the study was conducted in as well as methicillin resistance were important predictors of mortality. Therefore, for the second comparison, we matched each RCT to 4 observational studies involving patients from the same decade, describing the same population in detail (all SAB including methicillin-resistant S. aureus [MRSA] and methicillin-susceptible S. aureus [MSSA], only MRSA, or only MSSA), and reporting mortality at the same timepoint(s) as the RCT. For RCTs that were done in 1 country, at least 1 matched observational study was done in the same country.

We used descriptive analysis to compare RCTs to observational studies in terms of patient characteristics. All-cause mortality was analyzed as a single proportion using a generalized linear mixed model where the proportion underwent logit transformation [2]. This random-effects model estimated the pooled mortality for RCTs, matched observational studies, and all observational studies separately for all SAB, only MRSA, and only MSSA at the various timepoints that were used in the RCT. Heterogeneity was described using the T² method. Analysis was performed using the statistical software R version 3.6.3.

RESULTS

This secondary analysis included 7 RCTs [3–9]. For the RCT on trimethoprim-sulfamethoxazole for MRSA infections [9], we used the subgroup analysis results that included only patients with MRSA bacteremia. Notably, the RCT on daptomycin in SAB [10] was excluded because it described mortality at 42 days after end of therapy instead of a fixed timepoint relative to the first positive blood culture. The reported mortality was 26/335 (7.8%) in this study [10].

In the RCTs, the proportion of the screened population that was eligible and randomized to interventions ranged from 3% [5] to 35% [3]. The proportion of the screened population excluded due to death or being moribund ranged from 3% [5] to 11% [8]. The time limit from culture collection to enrollment ranged from 2 [4] to 7 [6] days. The inclusion and exclusion criteria for each RCT, the comparisons of the RCTs versus matched observational studies, and a description of all studies are included in Supplementary Materials 1, Supplementary Tables 1–3, respectively. All 7 RCTs did not show a significant difference in mortality between the treatment arms, so the mortality of all patients in trials was pooled.

The comparison of study and patient characteristics as well as pooled mortality rates over different timepoints in the RCTs, matched observational studies, and all observational studies are described in Table 1. RCTs and observational studies had similar mean age and sex distribution. However, the proportion of complicated bacteremia was higher in RCTs. Yet at all timepoints, for all types of bacteremia, and for matched and non-matched observational studies, the pooled mortality rates in RCTs were consistently and importantly lower. Whereas most RCTs recruited patients between days 2 and 7, the pooled mortality rates within days 2 and 7 in observational studies were 8.5% and 10.6% respectively for all SAB (Supplementary Materials 1, Supplementary Table 4).

Table 1.

Comparison of RCT and Observational Studies

	RCTs (N = 7)	Matched Observational (N = 28)	All Observational (N = 341)
Funding
Pharmaceutical	2 (29%)	1 (4%)	16 (5%)
Public funding	5 (71%)	16 (57%)	142 (42%)
Not funded	0 (0%)	10 (36%)	132 (39%)
Not specified	0 (0%)	1 (4%)	51 (15%)
Multicenter	7 (100%)	13 (46%)	109 (32%)
Resistance profile
MRSA and MSSA	2 (29%)	8 (29%)	215 (63%)
MSSA only	2 (29%)	8 (29%)	36 (11%)
MRSA only	3 (43%)	12 (43%)	90 (26%)
Sample size, median (IQR)	116 (98, 367)	311 (243, 618)	245 (130, 521)
Mean age reported, median (IQR)	58 (57, 61)	61 (59, 68)	61 (57, 65)
Female proportion, median (IQR)	35% (34%, 35%)	36% (34%, 38%)	37% (34%, 40%)
Median CCI, median (IQR)	4.0 (2.75, 5.0)	3.0 (2.0, 5.0)	3.0 (2.0, 3.0)
Median Pitt bacteremia score, median (IQR)	2.0 (n = 1)	1.0 (1.0, 1.75)	1.0 (1.0, 1.5)
Proportion of infections with the following characteristics
Nosocomial, median (IQR)	27% (23%, 36%)	36% (31%, 56%)	50% (33%, 68%)
Osteoarticular infections, median (IQR)	12% (12%, 16%)	18% (9%, 24%)	10% (6%, 15%)
Skin and soft tissue infections, median (IQR)	23% (22%, 25%)	17% (11%, 22%)	16% (12%, 22%)
Lung infections, median (IQR)	7% (6%, 14%)	12% (8%, 14%)	11% (7%, 16%)
Infective endocarditis, median (IQR)	12% (6%, 16%)	8% (5%, 15%)	7% (4%, 11%)
Complicated bacteremia, median (IQR)	73% (70%, 76%)	40% (28%, 86%)	48% (31%, 62%)
Proportion of patients who had the following interventions
ID consultation, median (IQR)	97% and 100% (n = 2)	76% (65%, 90%)	72% (51%, 82%)
Echocardiography, median (IQR)	100% (97%, 100%)	67% (54%, 71%)	54% (43%, 70%)
Mortality in studies on all SAB (MRSA and MSSA)
1-month pooled mortality (95% CI)	8.1% (6.5%–10.1%) T2 < 0.0001	17.7% (13.1%–23.4%) T2 = 0.2042	18.7% (17.5%–19.9%) T2 = 0.1697
3-month pooled mortality (95% CI)	14.8% (12.4%–17.5%) Single study	26.4% (23.1%–30.1%) T2 = 0.0434	26.4% (23.2%–29.8%) T2 = 0.3158
6-month pooled mortality (95% CI)	21.9% (15.3%–30.2%) Single study	27.3% (23.2%–31.8%) T2 = 0.0325	25.5% (18.8%–33.7%) T2 = 0.3012
Mortality in studies on only MRSA
2-week pooled mortality (95% CI)	7.6% (5.2%–10.9%) Single study	13.8% (8.2%–22.4%) Single study	20.7% (15.4%–27.3%) T2 = 0.3082
1-month pooled mortality (95% CI)a	21.9% (16.0%–29.2%) T2 < 0.0001	26.7% (20.9%–33.5%) T2 = 0.2521	26.0% (24.2%–27.9%) T2 = 0.1833
3-month pooled mortality (95% CI)a	18.3% (14.8%–22.4%) T2 < 0.0001	30.4% (26.8%–34.3%) T2 = 0.0188	35.7% (29.9%–41.9%) T2 = 0.1689
Mortality in studies on only MSSA
1-month pooled mortality (95% CI)	13.8% (11.0%–17.2%) T2 < 0.0001	17.4% (14.7%–20.5%) T2 = 0.0597	17.1% (15.5%–18.8%) T2 = 0.1680
3-month pooled mortality (95% CI)	19.0% (15.7%–22.7%) T2 < 0.0001	24.0% (20.2%–28.3%) T2 = 0.0624	23.9% (20.0%–28.4%) T2 = 0.2260

Baseline characteristics of RCT and matched observational studies are presented as dot plots in Supplementary Materials 1, Supplementary Figure 1–7.

Abbreviations: CCI, Charlson comorbidity index; CI, confidence interval; ID, infectious diseases; IQR, interquartile range; MRSA, methicillin-resistant S. aureus; MSSA, methicillin-susceptible S. aureus; RCT, randomized controlled trials.

The pooled mortality 3-month mortality was lower than the 1-month mortality because the estimates were based on different RCTs. One RCT reported 1-month mortality but not 3-month mortality, whereas 2 other RCTs reported 3-month mortality but not 1-month mortality.

DISCUSSION

There may be several reasons for the identified mortality gap between observational studies and RCTs. First, all included RCTs had relatively stringent eligibility criteria and recruited only one-third of the screened SAB population at best. Second, almost 10% of SAB patients were moribund or died such that they could not be included. In other words, RCTs select for patients who survive long enough to be assessed for eligibility, give consent, be randomized, and receive the allocated interventions. This leads to important bias given that SAB has a significant early mortality. All randomized controlled trials in this analysis recruited microbiologically confirmed SAB cases, so there was always immortal time built into the trial design that could not be eliminated. Finally, the protocolized care in terms of co-interventions in RCTs may be better than what can be achieved in everyday clinical practice. For example, infectious diseases consultation and echocardiography were performed in almost all patients in RCTs that recorded these quality markers, which were done much less frequently in observational studies.

These study findings have important implications for both researchers and clinicians. When designing clinical trials powered on mortality, mortality estimates should be taken from similar RCTs or from observational studies accounting for the conditional probability of mortality by the time of enrollment. That is, if an RCT is enrolling patients on day 7, mortality estimates based on observational data should exclude patients who died before day 7. This way, these trials can be adequately powered for plausible effect sizes. When applying RCT results to bedside care, clinicians should be aware of the possibility of a lower treatment effect than what was observed in RCTs, because the RCT excluded a proportion of the general SAB population with a poorer prognosis who were likely to die early regardless of the intervention received. The magnitude of this difference will depend on how the RCT patient characteristics and management align with patients that the clinicians would like to apply the intervention to in real world settings. In a systematic review that compared the treatment effect in RCTs versus observational studies, although not statistically significant, the pooled ratio of odds ratio was 1.08 (95% confidence interval [CI] .96 to 1.22) favoring larger effect size in RCTs [11].

A limitation of this study was the comparison across different studies and study designs, which extrapolated from an RCT to an observational study under different settings. The ideal comparison might be between included and excluded patients within the same trial, but the outcomes of excluded patients are rarely measured or reported. A prospective registry approach paired with an RCT would be highly valuable. To our knowledge, only 1 study on S. aureus infections did such a comparison [12]. This study compared outcomes of excluded and included patients with respect to a trial of trimethoprim-sulfamethoxazole for severe MRSA infections [12]. The study found that excluded patients were sicker and had worse outcomes even though the trial was intended to be pragmatic with few exclusion criteria [12], which supports the principles underpinning our study findings.

There is always room for improvement for future pragmatic trials, and this analysis suggests a pathway for improvement in terms of better generalizability. In the future, an emphasis on more pragmatic trials with minimal exclusion criteria to better represent the general SAB population, and a paired prospective registry may address the issue of generalizability. It would also be ideal for advances to be made in research ethics to expedite the consent and enrollment processes for trials. In particular, recruiting SAB patients earlier in the disease course will be essential to reduce pre-randomization death. These steps may help bridge the gap during the knowledge translation process from RCT to bedside care.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Acknowledgments. The authors thank Neera Bhatnagar for her guidance on search strategy. Study protocol, data set, and statistical analysis R code are available upon request from Anthony D Bai (email: [email protected])

Author contributions.

Conception and design: A. D. B., A. M. M., M.L., and T. C. L.

Abstract screening and data extraction: A. D. B., C. K. L. L., A. S. K., M. S., K. G., A. G., P. T., J. S., O. D. C., I. S., C. F., E. G. M., M. P. C., and T. C. L.

Data analysis: A. D. B. and G. B. L.

Writing of the article: A. D. B.

Revision of the manuscript: A. D. B., C. K. L. L., A. S. K., M. S., K. G., A. G., P. T., J. S., O. D. C., I. S., C. F., G. B. L., E. G. M., M. P. C., A. M. M., M. L., and T. C. L.

Disclaimer. The funding body had no role in the design of the study, data collection, analysis, interpretation of data or writing of the manuscript.

Financial support. This project was supported by the McMaster Medicine Specialty Resident and Fellows Research Grant.

Author notes

Potential conflicts of interest. M. P. C. reports grants from McGill Interdisciplinary Initiative in Infection and Immunity and grants from Canadian Institutes of Health Research during the conduct of the study; personal fees from GEn1E Lifesciences (as a member of the scientific advisory board) and personal fees from nplex biosciences (as a member of the scientific advisory board) and consulting fees from Astra Zaneca outside the submitted work. He is the co-founder of Kanvas Biosciences and owns equity in the company. In addition, M. P. C. has a patent Methods for detecting tissue damage, graft versus host disease, and infections using cell-free DNA profiling pending (Cornell Reference number 9401-01-US), and a patent Methods for assessing the severity and progression of severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) infections using cell-free DNA pending (Cornell Reference number 9561-01-US) and rapid identification of antimicrobial resistance and other microbial phenotypes using highly multiplexed fluorescence in situ hybridization (Cooley Reference: BEEB-001/00US 342794-2002). T. C. L. and E. G. M. receive research salary support from the Fonds de recherche du Québec – Santé. M. L. reports having served on advisory boards for Sanofi, Pfizer, Medicago, Merck, Seqirus, Paladin Labs; Pan Data Safety, and Monitoring Committees for Medicago, CanSino biologics, National Institutes of Health (NIH), the World Health Organization (WHO) EML Antibiotic Working Group. E. G. M. reports being co-investigator on the SNAP staph aureus platform trial CIHR outside of the submitted work. T. C. L. reports operating funds for other studies from Canadian Institutes of Health Research and McGill Interdisciplinary Institute Infection and Immunity. All other authors report no potential conflicts. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest. Conflicts that the editors consider relevant to the content of the manuscript have been disclosed.