Effectiveness of the 2023–2024 Formulation of the COVID-19 Messenger RNA Vaccine

Abstract

Background

The purpose of this study was to evaluate whether the 2023–2024 formulation of the coronavirus disease 2019 (COVID-19) messenger RNA vaccine protects against COVID-19.

Methods

Cleveland Clinic employees when the 2023–2024 formulation of the COVID-19 messenger RNA vaccine became available to employees were included. Cumulative incidence of COVID-19 over the following 17 weeks was examined prospectively. Protection provided by vaccination (analyzed as a time-dependent covariate) was evaluated using Cox proportional hazards regression, with time-dependent coefficients used to separate effects before and after the JN.1 lineage became dominant. The analysis was adjusted for the propensity to get tested, age, sex, pandemic phase when the last prior COVID-19 episode occurred, and the number of prior vaccine doses.

Results

Among 48 210 employees, COVID-19 occurred in 2462 (5.1%) during the 17 weeks of observation. In multivariable analysis, the 2023–2024 formula vaccinated state was associated with a significantly lower risk of COVID-19 before the JN.1 lineage became dominant (hazard ratio = .58; 95% confidence interval [CI] = .49–.68; P < .001), and lower risk but one that did not reach statistical significance after (hazard ratio = .81; 95% CI = .65–1.01; P = .06). Estimated vaccine effectiveness was 42% (95% CI = 32–51) before the JN.1 lineage became dominant, and 19% (95% CI = −1–35) after. Risk of COVID-19 was lower among those previously infected with an XBB or more recent lineage and increased with the number of vaccine doses previously received.

Conclusions

The 2023–2024 formula COVID-19 vaccine given to working-aged adults afforded modest protection overall against COVID-19 before the JN.1 lineage became dominant, and less protection after.

SARS-CoV-2, COVID-19, effectiveness, vaccine, prospective cohort study

The original messenger RNA (mRNA) coronavirus disease 2019 (COVID-19) vaccines were highly efficacious when examined in randomized clinical trials [1,2]. Vaccine effectiveness was subsequently confirmed in the real world outside of clinical trials [3,4], including among employees within our own healthcare system [5]. Although the causative agent of COVID-19, the severe acute respiratory syndrome coronavirus 2 (SARS-CoV-2) virus evolved over time for almost 2 years since the onset of the pandemic, those previously infected or vaccinated continued to have substantial protection against COVID-19 by virtue of natural or vaccine-induced immunity [6].

With the arrival of the Omicron variant in December 2021, previously infected or vaccinated individuals were less protected from COVID-19 [6]. Boosting with the original vaccine provided some protection against the Omicron variant [7,8], but the degree of protection was less than that against the pre-Omicron variants of SARS-CoV-2 [8]. Effectiveness of the vaccine steadily decreased as newer virus variants emerged and became the dominant circulating strains. Prior infection with an earlier lineage of the Omicron variant protected somewhat against subsequent infection with a subsequent lineage [9], but such protection appeared to wear off within a few months [10]. During the Omicron phase of the pandemic, protection from vaccine-induced immunity also decreased within a few months after vaccine boosting [8].

The waning protection afforded by the original vaccines spurred the development of newer vaccines that were expected to be more effective. On 31 August 2022, the United States Food and Drug Administration approved bivalent COVID-19 mRNA vaccines, which encoded antigens represented in the original vaccine as well as antigens representing the BA.4/5 lineages of the Omicron variant. Given a perception of an urgent need for updated vaccines, these vaccines were approved without demonstration of effectiveness in clinical studies. Subsequent studies found that the bivalent vaccines were about 29% effective overall in protecting against infection with SARS-CoV-2 when the Omicron BA.4/5 lineages were the predominant circulating strains [11, 12], but only 19% effective when the BQ lineages were dominant [11], and no longer effective by the time the XBB lineages became dominant [11]. In April 2023, the United States Centers for Disease Control and Prevention (CDC) updated its guidance on COVID-19 vaccination to consider all individuals above the age of 6 years to be “up to date” with COVID-19 vaccination only if they had received at least 1 dose of a COVID-19 bivalent vaccine [13]. Expectedly, when the XBB lineages were dominant, those up to date with bivalent vaccination were not at lower risk of COVID-19 than those not up to date [14].

On 11 September 2023, the Food and Drug Administration approved and authorized for emergency use updated COVID-19 vaccines made by Moderna TX Inc. and Pfizer-BioNTech Inc., which were formulated to more closely target currently circulating variants (2023–2024 formula) [15]. Both of these vaccines encode the spike protein of SARS-CoV2 Omicron variant lineage XBB.1.5 (Omicron XBB.1.5) [16]. On 12 September 2023, the CDC recommended these vaccines for everyone aged 6 months and older [17]. However, by the time these vaccines became available to the public, the XBB lineages were no longer the dominant circulating strains in many parts of the United States [18].

The purpose of this study was to evaluate whether the 2023–2024 formula COVID-19 vaccine protects against COVID-19.

METHODS

Study Design

This was a prospective cohort study conducted at the Cleveland Clinic Health System (CCHS) in the United States.

Patient Consent Statement

The study was approved by the Cleveland Clinic institutional review board as exempt research (institutional review board no. 22-917). A waiver of informed consent and waiver of Health Insurance Portability and Accountability Act authorization were approved to allow the research team access to the required data.

Setting

Cleveland Clinic has always given very high priority to employee access to COVID-19 testing and COVID-19 vaccination. The 2023–2024 formulation of the COVID-19 vaccine was available at Cleveland Clinic to all employees beginning 10 October 2023. This date was considered the study start date.

The mix of circulating variants of SARS-CoV-2 has evolved throughout the pandemic. The majority of SARS-CoV-2 infections in Ohio were caused by post-XBB lineages even at the start of the study; by the end of the study, almost all infections were caused by the JN.1 lineage (Supplementary Figure 1).

Participants

CCHS employees in employment at any Cleveland Clinic location in Ohio on 10 October 2023, the day the 2023–2024 formulation of the COVID-19 vaccine was available to employees at Cleveland Clinic, were included in the study. Those for whom age and gender were not available were excluded.

Variables

Covariates collected were age, sex, and job location, as described in our earlier studies [5–7, 11, 14]. Institutional data governance rules related to employee data limited our ability to supplement our dataset with additional clinical variables. Subjects were considered prepandemic hires if hired before 16 March 2020, the day COVID-19 testing became available in our institution, and pandemic hires if hired on or after that date.

Number of prior vaccine doses included those received before the study start date. Prior COVID-19 was defined as a positive nucleic acid amplification test (NAAT) for SARS-CoV-2 any time before the study start date. The date of infection for a prior episode of COVID-19 was the date of the first positive test for that episode of illness. A positive test more than 90 days following the date of a previous infection was considered a new episode of infection. The propensity to get tested for COVID-19 was defined as the number of COVID-19 NAATs done divided by the number of years of employment at CCHS during the pandemic, before the study start date.

The distribution of circulating variants in Ohio at any time were obtained from monitoring data from the CDC [18]. Supplementary Figure 1 shows how SARS-CoV-2 lineage proportions changed over time beginning 1 year before the study start date. The pandemic phase for prior episodes of COVID-19 (pre-Omicron, pre-XBB Omicron, and XBB Omicron or later) was defined by which variant/lineages accounted for more than 50% of infections in Ohio at the time of the infection [18].

Outcome

The study outcome was time to COVID-19, the latter defined as a positive NAAT for SARS-CoV-2 any time after the study start date. Outcomes were followed until 5 February 2024, allowing for evaluation of outcomes up to 17 weeks from the study start date.

Statistical Analysis

A Simon-Makuch hazard plot [19] was created to compare the cumulative incidence of COVID-19 in the vaccinated and nonvaccinated states with respect to the 2023–2024 formula COVID-19 vaccine, by treating such vaccination as a time-dependent covariate. Individuals were considered vaccinated 7 days after receipt of a single dose of the 2023–2024 formula COVID-19 vaccine. Subjects whose employment was terminated during the study period before they had COVID-19 were censored on the date of termination. Curves for the nonvaccinated state were based on data, whereas the vaccination status of subjects, with respect to the 2023–2024 formula COVID-19 vaccine, remained “nonvaccinated.” Curves for the vaccinated state were based on data from the date the vaccination status changed to “vaccinated.”

A Cox proportional hazards regression model was fit to examine the association of various variables with time to COVID-19. Vaccination with the 2023–2024 formula COVID-19 vaccine was included as a time-dependent covariate [20]. The study period was divided into 2 phases demarcated by when the JN.1 lineage became the dominant circulating strain in Ohio (accounted for >50% of infections). Time-dependent coefficients were used to separate out the effects of the 2023–2024 formula COVID-19 vaccine before and after the JN.1 lineage became dominant. The possibility of multicollinearity in the models was evaluated using variance inflation factors. The proportional hazards assumption was checked using log(-log(survival)) versus time plots. Vaccine effectiveness (VE) was calculated from the hazard ratios (HRs) for 2023–2024 formula COVID-19 vaccination in the multivariable model using the formula VE = 1 – HR.

The analysis was performed by N. K. S. and A. S. N. using the survival package and R version 4.2.2 [20–22].

RESULTS

Of the 48 210 employees included in the study, 520 had received the 2023–2024 formula COVID-19 vaccine outside the institution 7 days or more before the study start date, and 1818 (3.8%) were censored during the study because of termination of employment. By the end of the study, 7978 (17%) had received the 2023–2024 formula COVID-19 vaccine, which was the Pfizer vaccine in 7104 (89%). Of the 6486 subjects who had not previously been vaccinated, 246 (4%) received the latest vaccine. Altogether, 2462 employees (5.1%) acquired COVID-19 during the 17 weeks of the study.

Baseline Characteristics

Table 1 shows the characteristics of subjects included in the study. Notably, this was a relatively young population, with a mean age of 42 years. Among these, 21 716 (45%) had previously had a documented episode of COVID-19 and 16 661 (35%) had previously had an Omicron variant infection. A total of 41 748 subjects (87%) had previously received at least 1 dose of vaccine, 39 844 (83%) had received at least 2 doses, and 43 350 (90%) had been previously exposed to SARS-CoV-2 by infection or vaccination.

Table 1.

Baseline Characteristics of the Included 48 210 Employees of Cleveland Clinic in Ohio, and Features of Those who Did and Did not Receive the 2023–2024 Formulation COVID-19 Vaccine by the end of the Study

Characteristics	All Subjects (n = 48 210)	Vaccinateda by the End of the Study (n = 7978)	Not Vaccinateda by the End of the Study (n = 40 232)	P
Age in y, mean (SD)	42.0 (13.4)	47.6 (13.9)	40.9 (13.0)	<.001
Sex				<.001
Female	35 959 (74.6)	5421 (67.9)	30 538 (75.9)
Male	12 251 (25.4)	2557 (32.1)	9694 (24.1)
Location				<.001
Cleveland Clinic Main	19 935 (41.4)	4476 (56.1)	15 459 (38.4)
Cleveland area regional hospitals	12 289 (25.5)	1450 (18.2)	10 839 (26.9)
Ambulatory centers	8922 (18.5)	1198 (15.0)	7724 (19.2)
Cleveland Clinic Akron	4075 (8.5)	348 (4.4)	3727 (9.3)
Administrative centers	1779 (3.7)	377 (4.7)	1402 (3.5)
Cleveland Clinic Medina	1210 (2.5)	129 (1.6)	1081 (2.7)
Hire cohort				<.001
Prepandemic	28 030 (58.1)	5404 (67.7)	22 626 (56.2)
Pandemic	20 180 (41.9)	2574 (32.3)	17 606 (43.8)
Pandemic phase during which most recent infection occurred				<.001
Not previously known to be infected	26 494 (55.0)	4526 (56.7)	21 968 (54.6)
Pre-Omicron	5055 (10.5)	472 (5.9)	4583 (11.4)
Omicron pre-XBB	13 499 (28.0)	2360 (29.6)	11 139 (27.7)
Omicron XBB or later	3162 (6.6)	620 (7.8)	2542 (6.3)
Days since most recent infection, mean (SD)	544 (284)	478 (260)	557 (286)	<.001
Prior vaccination history				<.001
Monovalent and bivalent vaccines	12 264 (25.4)	5876 (73.7)	6388 (15.9)
Bivalent vaccine only	379 (<1)	111 (1.4)	268 (<1)
Monovalent vaccine only	28 192 (58.5)	1722 (21.6)	26 470 (65.8)
Non-mRNA vaccine only	913 (1.9)	23 (<1)	890 (2.2)
No vaccine	6462 (13.4)	246 (3.1)	6216 (15.5)
Number of prior vaccine doses				<.001
0	6462 (13.4)	246 (3.1)	6216 (15.5)
1	1904 (3.9)	134 (1.7)	1770 (4.4)
2	13 381 (27.8)	303 (3.8)	13 078 (32.5)
3	14 106 (29.3)	1287 (16.1)	12 819 (31.9)
>3	12 357 (25.6)	6008 (75.3)	6349 (15.8)
Days since most recent vaccine, mean (SD)	597 (232)	377 (197)	648 (209)	<.001
Days since proximate SARS-CoV-2 exposureb, mean (SD)	513 (239)	339 (188)	551 (232)	<.001

Data are presented as no. (%) unless otherwise indicated.

Abbreviations: COVID-19, coronavirus disease 2019; mRNA, messenger RNA; SARS-CoV-2, severe acute respiratory syndrome coronavirus 2; SD, standard deviation.

^aWith the 2023–2024 formulation vaccine.

^bBy infection or vaccination.

Effectiveness of the COVID-19 mRNA Vaccine (2023–2024 Formulation)

The cumulative incidence of COVID-19 was lower in the 2023–2024 formula vaccinated state compared with the nonvaccinated state in an unadjusted analysis (Figure 1).

Figure 1.

Simon-Makuch plot comparing the cumulative incidence of COVID-19 for the vaccinated and nonvaccinated states with respect to the 2023–2024 formulation COVID-19 vaccine. Day 0 was 10 October 2023, the day the 2023 formulation COVID-19 vaccine began to be offered to employees. Individuals with recent past infections do not contribute to the denominator until it has been at least 90 d since their prior infection. Point estimates and 95% confidence intervals are jittered along the x-axis to improve visibility. Variant proportions are based on data from the CDC, considered to be in the middle of the week reported, grouped into pre-XBB, XBB, HV.1, EG.5, JN.1, and other lineages, and presented as an area plot with values extrapolated for day 0 using linear regression with data for the weeks immediately before and after day 0. Abbreviations: CDC, Centers for Disease Control and Prevention; COVID-19, coronavirus disease 2019.

In a multivariable Cox proportional hazards regression model, adjusted for propensity to get tested for COVID-19, age, sex, hire cohort, number of COVID-19 vaccine doses before the study start, and epidemic phase when the last prior COVID-19 episode occurred, vaccination with the 2023–2024 formula of the COVID-19 vaccine provided some protection against COVID-19 in the first 17 weeks after the vaccine became available at our institution, better protection before the JN.1 lineage became dominant (HR = .58; 95% CI = .49–.68; P < .001), but less protection after (HR = .81; 95% CI = .65–.1.01; P= .07). Point estimates and 95% CIs for HRs for the variables included in the unadjusted and adjusted Cox proportional hazards regression models are shown in Table 2. The calculated overall vaccine effectiveness from the model was 42% (95% CI = 32–51) before the JN.1 lineage became dominant, and 19% (95% CI = −1–35) after.

Table 2.

Unadjusted and Adjusted Associations With Time to COVID-19

Variables	Unadjusted HR (95% CI)	P	Adjusted HR (95% CI)a	P
2023 formulation vaccinated stateb
Before JN.1 lineage became dominant	.70 (.60–.82)	<.001	.58 (.49–.68)	<.001
After JN.1 lineage became dominant	1.000 (.81–1.23)	.99	.81 (.65–1.01)	.06
Propensity to get tested for COVID-19	1.02 (1.02–1.03)	<.001	1.02 (1.02–1.03)	<.001
Age	1.001 (.998–1.004)	.38	1.000 (.997–1.003)	.91
Male sexc	.74 (.67–.81)	<.001	.75 (.68–.83)	<.001
Prepandemic hired	1.03 (.95–1.12)	.47	.87 (.79–.95)	.002
Last prior infection phase (ref: No known prior infection)
Pre-Omicron	1.25 (1.10–1.42)	<.001	1.23 (1.08–1.40)	<.001
Omicron pre-XBB	1.31 (1.20–1.43)	<.001	1.21 (1.10–1.32)	<.001
Omicron XBB or later	.50 (.39–.65)	<.001	.44 (.34–.57)	<.001
Number of prior vaccine doses (ref: 0)
1	1.96 (1.51–2.55)	<.001	1.99 (1.53–2.58)	<.001
2	2.00 (1.68–2.38)	<.001	1.95 (1.63–2.33)	<.001
3	2.49 (2.10–2.96)	<.001	2.60 (2.18–3.10)	<.001
>3	2.43 (2.04–2.89)	<.001	3.00 (2.50–3.60)	<.001

Abbreviations: CI, confidence interval; COVID-19, coronavirus disease 2019; HR, hazard ratio.

^aFrom a multivariable Cox-proportional hazards regression model with the 2023–2024 formulation vaccinated state treated as a time-dependent covariate, and time-dependent coefficients used to separate effects before and after the JN.1 lineage became dominant.

^bTime-dependent covariate.

^cReference is female sex.

^dReference is pandemic hire.

The multivariable analysis also found that those who had recently had a prior infection with an XBB or more recent lineage of the virus had a lower risk of COVID-19 and that the greater the number of vaccine doses previously received the higher the risk of COVID-19. The reference level for the latter covariate was 0 doses. Significant associations with higher doses remained (all P values <.001) even if “0 or 1 dose” or “0, 1, or 2 doses” were used as reference levels.

DISCUSSION

This study found that the 2023–2024 formula COVID-19 mRNA vaccine was about 42% effective overall in protecting against infection with SARS-CoV-2 before the JN.1 lineage became dominant, and only 19% effective after. This formulation of the vaccine was designed to target the XBB lineages of the Omicron variant, and the vaccine was still effective even though the majority of infections occurring in the community were caused by post-XBB lineages of the virus by the time this formulation became available at our institution and the study was conducted. The finding of lower effectiveness of the vaccine after the JN.1 lineage became dominant could be because the vaccine is less effective against the JN.1 lineage, but it is also possible that although the vaccine remains effective against this lineage the effect of the vaccine wears off within a couple of months. How long the vaccine remains effective remains to be seen.

The strengths of our study include its large sample size and its conduct in a healthcare system where a very early recognition of the critical importance of maintaining an effective workforce during the pandemic led to devotion of resources to have an accurate accounting of who had COVID-19, when COVID-19 was diagnosed, who received COVID-19 vaccines, and when they received them. The study methodology, treating vaccination with the 2023–2024 formulation of the COVID-19 vaccine as a time-dependent covariate, allowed for determining vaccine effectiveness in real time. Adjusting for hire cohort (prepandemic vs pandemic) in the multivariable analysis would have mitigated against bias that might arise from possible incomplete information on prior infection and prior vaccination among the pandemic hires compared to the prepandemic hires.

The study has several limitations. Individuals with unrecognized prior infection would have been misclassified as previously uninfected. Because prior infection protects against subsequent infection, such misclassification would have resulted in underestimating the protective effect of the vaccine. However, there is little reason to suppose that prior infections would have been missing in the vaccinated and nonvaccinated states at disproportionate rates. There might be concern that those who chose to receive the newest vaccine might have been more worried about infection and might have been more likely to have gotten tested when they had symptoms, thereby disproportionately detecting more incident infections among those who received the vaccine. The potential for risk of bias from this effect was mitigated by adjusting for the propensity of an individual to get tested for COVID-19. The widespread availability of home testing kits might have reduced detection of incident infections. This potential effect should be somewhat mitigated in our healthcare cohort because one needs a NAAT to get paid time off, providing a strong incentive to get a NAAT if one tests positive at home. It is also possible that nonsalaried and remote workers may have had less of an incentive to follow-up a positive home test with a NAAT. Even if one assumes that some individuals chose not to follow up on a positive home test result with a NAAT or chose not to get tested at all when they had symptoms, it is very unlikely that individuals would have chosen to pursue NAAT after receiving the newest vaccine more so than before receiving the vaccine, at rates disproportionately enough to affect the study's findings. We were unable to distinguish between symptomatic and asymptomatic infections and had to limit our analyses to all detected infections. The number of severe illnesses was too small to examine as an outcome. Last, our study was done in a healthcare population that included no children and few elderly subjects; most of the study subjects would not have been immunocompromised.

The effectiveness of the vaccine was lower than one might have expected. A possible explanation for a weaker than expected vaccine effectiveness is that a proportion of the population may have had recent prior asymptomatic COVID-19. About one third of SARS-CoV-2 infections have been estimated to be asymptomatic in studies that have been done in different places at different times [23–25]. If so, protection from the newest vaccine may have been masked somewhat because those with unrecognized recent COVID-19 may have already been somewhat protected against COVID-19 by virtue of natural immunity. Another explanation is that the COVID-19 vaccine becomes increasingly less effective as circulating variants become increasingly different from the variant against which the vaccine was developed [11]. By the time the latest vaccine became available to the public, the majority of circulating strains were lineages that had appeared after the vaccine was developed [18]. By the time the JN.1 lineages became dominant, the protection afforded against infection by the 2023–2024 formulation was low.

The study also adds to the increasing evidence that a higher number of prior vaccine doses is associated with a higher risk of COVID-19 [7, 11, 12, 14, 26, 27]. It is possible that the association of increasing risk of COVID-19 with increasing number of prior vaccine doses occurred because of limitations of our dataset. On the other hand, immune imprinting from prior exposure to different antigens in a prior vaccine (original antigenic sin) [27, 28], and class switch toward noninflammatory spike-specific IgG4 antibodies after repeated SARS-CoV-2 mRNA vaccination [29], have been suggested as possible mechanisms for why those previously vaccinated may be at higher risk of infection. The original antigenic sin hypothesis has been studied much more extensively in influenza in which there is a better understanding of beneficial and potentially detrimental effects of the phenomenon [30], and it has been reasonably hypothesized that sequential COVID-19 vaccination of low-risk patients could be detrimental [31]. Caution has been advised about trying to prevent all symptomatic infections in healthy young individuals by boosting them with vaccines targeting strains that might no longer be in circulation in a few months [32], or indeed even by the time the vaccine becomes widely available.

In conclusion, this study found an overall modest protective effect of the 2023–2024 formula COVID-19 vaccine against SARS-CoV-2 infection before the JN.1 lineage became dominant but a lower level of protection after, while also finding a higher risk of COVID-19 among those with a higher number of prior vaccine doses.

Supplementary Data

Supplementary materials are available at Clinical Infectious Diseases online. Consisting of data provided by the authors to benefit the reader, the posted materials are not copyedited and are the sole responsibility of the authors, so questions or comments should be addressed to the corresponding author.

Notes

Author Contributions. N. K. S.: Conceptualization, methodology, validation, investigation, data curation, software, formal analysis, visualization, writing – original draft preparation, writing – reviewing and editing, supervision, project administration. P. C. B.: Resources, investigation, validation, writing – reviewing and editing. A. S. N.: Methodology, formal analysis, visualization, validation, writing – reviewing and editing. S. M. G.: Project administration, resources, writing – reviewing and editing.

Author notes

Potential conflicts of interest. The authors: No reported conflicts of interest. All authors have submitted the ICMJE Form for Disclosure of Potential Conflicts of Interest.