SMART-COP: A Tool for Predicting the Need for Intensive Respiratory or Vasopressor Support in Community-Acquired Pneumonia

Abstract

Background. Existing severity assessment tools, such as the pneumonia severity index (PSI) and CURB-65 (tool based on confusion, urea level, respiratory rate, blood pressure, and age <65 years), predict 30-day mortality in community-acquired pneumonia (CAP) and have limited ability to predict which patients will require intensive respiratory or vasopressor support (IRVS).

Methods. The Australian CAP Study (ACAPS) was a prospective study of 882 episodes in which each patient had a detailed assessment of severity features, etiology, and treatment outcomes. Multivariate logistic regression was performed to identify features at initial assessment that were associated with receipt of IRVS. These results were converted into a simple points-based severity tool that was validated in 5 external databases, totaling 7464 patients.

Results. In ACAPS, 10.3% of patients received IRVS, and the 30-day mortality rate was 5.7%. The features statistically significantly associated with receipt of IRVS were low systolic blood pressure (2 points), multilobar chest radiography involvement (1 point), low albumin level (1 point), high respiratory rate (1 point), tachycardia (1 point), confusion (1 point), poor oxygenation (2 points), and low arterial pH (2 points): SMART-COP. A SMART-COP score of <3 points identified 92% of patients who received IRVS, including 84% of patients who did not need immediate admission to the intensive care unit. Accuracy was also high in the 5 validation databases. Sensitivities of PSI and CURB-65 for identifying the need for IRVS were 74% and 39%, respectively.

Conclusions. SMART-COP is a simple, practical clinical tool for accurately predicting the need for IRVS that is likely to assist clinicians in determining CAP severity.

Community-acquired pneumonia (CAP) is the leading infectious cause of death in the United States [1]. It is responsible for ∼1 million admissions per annum, with health care expenditure in excess of $10 billion [2]. Severity assessment tools have been developed to help guide the sites of care for patients with CAP and, in particular, to identify patients whose condition can be managed safely at home. The most popular of these tools are the pneumonia severity index (PSI) [3] and CURB-65 (a tool based on confusion, urea level, respiratory rate, blood pressure, and age <65 years) [4]. Both were developed from statistical analyses of features associated with 30-day mortality. The presence of such features is converted into a score that indicates the patient's risk of death and can be used to guide the choice of inpatient versus outpatient care. Although 30-day mortality is clearly an important outcome, the vast majority of patients who die of CAP are elderly persons with multiple comorbidities [5, 6]. When such patients are admitted to the hospital, aggressive treatment in the intensive care unit (ICU) is often considered inappropriate, given their poor quality of life and prognosis [7]. Thus, tools that are accurate in predicting mortality are less accurate for identifying patients likely to benefit from admission to the ICU [8–12].

The ability to predict which patients will require ICU admission can be difficult because clinicians both overestimate and underestimate the severity of CAP [13, 14]. Patients who require ICU admission consume a large proportion of health care expenditure [2, 15]. Early recognition of such patients could improve outcomes, avoid inappropriate nonadmissions, and potentially lead to a shorter length of ICU stay.

Because criteria for ICU admission vary both between hospitals and between countries, we aimed to assess features specifically associated with receipt of intensive respiratory or vasopressor support (IRVS; i.e., invasive or noninvasive mechanical ventilation or infusions of vasopressors for blood pressure support), rather than simple ICU admission, because these are likely to be objective markers of CAP severity across institutions and health care systems.

The Australian Community-Acquired Pneumonia Study (ACAPS) was a prospective, multicenter, observational study that assessed the etiology, severity markers, and treatment outcomes of a large population of patients with CAP defined by strict criteria [16]. We used these data to develop a new tool to identify patients with CAP who require IRVS.

Methods

Study design and setting. Patient recruitment and the inclusion and exclusion criteria have been described elsewhere [17]; the only difference for this aspect of the study is that pregnant patients were excluded from analysis. The following patient details were recorded: demographic characteristics, comorbid illnesses, initial vital signs, and the various investigational results required to calculate PSI [3] and CURB-65 [4] scores. In calculating the PSI score, we allowed the use of pulse oximetry scores ⩽90% to obtain the 10 points for hypoxia [3] and the use of pH from a venous blood sample (30 points for venous pH <7.30) if arterial puncture was thought to be inappropriate, because this correlates with arterial acidosis [18]. On the basis of CURB-65 scores, patients were classified into CURB-65 group 1 (scores 0–1), group 2 (score 2), and group 3 (scores 3–5) [4]. In addition, we recorded other comorbidities (smoking status, asthma, chronic obstructive pulmonary disease [COPD], diabetes mellitus, alcohol abuse, injection drug use, neuromuscular conditions, epilepsy, or dementia), oral antibiotic and corticosteroid use before hospitalization, vital signs at the time of arrival and the worst results (e.g., highest respiratory rate and lowest blood pressure) in the first 24 h, other initial laboratory results (including WBC count, serum albumin level, erythrocyte sedimentation rate, and C-reactive protein level), and results of urinary antigen tests for Legionella infection (Binax). The number of pulmonary lobes involved and the presence of pleural effusions were assessed on chest radiography. Blood culture specimens were obtained before parenteral administration of antibiotics.

In the hospital, patients were assessed for time to clinical stability [19], admission to the ICU (including the high dependency unit or coronary care unit), and length of hospital stay. In particular, we recorded the receipt of IRVS as a more objective marker of severity than simple ICU admission. At 4–6 weeks after admission, patients were assessed for cure and for 30-day mortality.

Statistical analysis and development of the severity prediction tool. Univariate analysis was performed using logistic regression to explore associations between patient characteristics and clinical features at the time of admission and the risk of subsequent receipt of IRVS. For multivariate analyses, a preliminary step of exploratory analysis was pursued in which the cohort was stratified on age ⩽50 years; separate logistic regression models were developed for older and younger patients, and a small number of different cutoff points were considered for each covariate. This was repeated with stratification on age <40 years. On the basis of these exploratory analyses, definitions of high respiratory rate and hypoxia differed for patients aged >50 years and those aged ⩽50 years, and this concurred with a priori clinical expectations. For patients aged ⩽50 years, tachypnea was defined as a respiratory rate <25 breaths/min; for patients aged >50 years, it was defined as <30 breaths/min. Hypoxia was defined as partial pressure of oxygen (PaO₂) <70 mm Hg, PaO₂ divided by the fraction of inspired oxygen (PaO₂/FiO₂) <333, or arterial oxygen saturation (SpO₂) ⩽93% for patients aged ⩽50 years and as PaO₂ <60 mm Hg, PaO₂/FiO₂ <250, or SpO₂ ⩽90% for patients aged >50 years. It was assumed that patients without arterial pH measurements had a normal value [3].

Multivariate logistic regression analyses of the features associated with receipt of IRVS were performed using sex, age ⩽50 years, and all variables that had P<.2 on univariate analysis, had prevalence >5%, were observed in at least 95% of patients, and would be easily and rapidly ascertainable by hospital clinicians. A backwards stepwise selection procedure was used with removal of variables that had P>.1 and reinclusion of variables that had P<.05. As a strategy to avoid overfitting, 1000 bootstrap replications of the selection procedure were performed, and only variables present in at least 60% of replications were retained in the final multivariate model [20]. A second model was obtained by retaining in the model only those variables that would be easily ascertained in primary care, without laboratory investigations.

The logistic regression model formula was

where X₁, X₂, and so forth were variables, such as hypoxia and tachycardia, and b₁, b₂, and so forth were the corresponding “beta” coefficients. The 2 final models were simplified by assigning a score of 1 point to variables with a beta coefficient ⩽1.2 (corresponding to an OR of 3.2) and 2 points to variables with a beta coefficient >1.2 (the so-called minor and major criteria, respectively). All statistical calculations were performed using Stata, version 9 (Stata Corp.).

External validation. We calculated scores for the new prediction rules for a total of 7464 patients from 5 existing databases, whose patient enrollment characteristics are shown in table 1 [3, 21–24]. Area under the receiver operating characteristic curve (AUC) and the Hosmer-Lemeshow goodness-of-fit statistic were calculated to assess the discriminability and calibration of the derived tools in predicting the need for IRVS.

Table 1

Characteristics of validation cohort studies.

Results

Patient population for derivation. Approximately 2500 patients were assessed, and 882 episodes of CAP involving 862 patients were included. The main reasons for exclusion were normal chest radiography, receipt of parenteral antibiotics before obtainment of blood culture specimens, hospitalization within the preceding 2 weeks, or suspected aspiration. Patient demographic characteristics, clinical features (including PSI and CURB-65 scores), and sites of enrollment are shown in table 2.

Table 2

Baseline characteristics of patients experiencing episodes of community-acquired pneumonia (CAP) in the Australian CAP Study.

Need for IRVS and 30-day mortality. Admission to the ICU occurred in 118 (13.4%) of 882 episodes, and IRVS was required in 91 (10.3%) of 882 episodes; of the 91 patients involved, 40 (44.0%) were intubated, 41 (45.1%) received noninvasive ventilation, and 38 (41.8%) received vasopressor support. Of the 91 patients who received IRVS, 53 (58.2%) were admitted to the ICU directly from the emergency department, whereas the remaining 38 (41.8%) were initially admitted to general wards and later were transferred to the ICU.

Overall, the 30-day mortality rate was 5.7% (50 deaths in 882 episodes), and 14 (15.4%) of the 91 patients who required IRVS died. Thus, only 14 (28.0%) of the 50 patients who died within 30 days after hospital admission had been admitted to the ICU; all received IRVS. The remaining patients who died were all designated “not for resuscitation” (NFR) and had active treatment withdrawn.

The performance of the PSI and CURB-65 tools in predicting the need for IRVS and 30-day mortality is shown in table 3. PSI classes IV and V together predicted 67 (73.6%) of the 91 patients who received IRVS. However, 9 (9.9%) of the 91 patients were in PSI classes I and II, and 15 (16.5%) were in PSI class III. In comparison, CURB-65 group 3 predicted the need for IRVS in 35 (38.5%) of the 91 patients, whereas 30 (33.0%) and 26 (28.6%) of the 91 patients were in CURB-65 groups 1 and 2, respectively.

Table 3

Relationship between the severity of community-acquired pneumonia scored using 2 prediction tools and the receipt of intensive respiratory or vasopressor support (IRVS) and 30-day mortality.

Features associated with IRVS and development of SMART-COP. The features that were associated with IRVS in univariate and multivariate analyses are shown in table 4. In the multivariate analyses of the 849 patients who had complete data available for all variables considered, age, sex, asthma, COPD, smoking status, injection drug use, immunosuppression, low diastolic blood pressure, sodium level, WBC count, and elevated urea level were excluded. On the basis of their beta coefficients, hypoxia, hypotension, and low arterial pH had the strongest associations with IRVS and were major criteria, whereas confusion, tachycardia, tachypnea, multilobar chest radiography involvement, and low serum albumin level were minor criteria. Initials of the 8 features that were associated with the need for IRVS in the final multivariate model were summarized in the mnemonic “SMART-COP” (systolic blood pressure, multilobar chest radiography involvement, albumin level, respiratory rate, tachycardia, confusion, oxygenation, and arterial pH) (figure 1).

Table 4

Univariate and multivariate analyses of features associated with receipt of intensive respiratory or vasopressor support.

Figure 1

Flow chart for the use of SMART-COP. BP, blood pressure; bpm, beats/min; br, breaths; CXR, chest radiography (x-ray); RR, respiratory rate; yo, years old.

A modified version of SMART-COP that is suitable for use in primary care settings was created by removing from SMART-COP the need for measurements of albumin level, arterial pH, and PaO₂. The major criteria (2 points each) in this primary care tool are low systolic blood pressure and hypoxia (based on pulse oximetry results) (figure 1), and the minor criteria (1 point each) are confusion, tachycardia, tachypnea, and multilobar chest radiography involvement. The first initials of these features were summarized in the mnemonic “SMRT-CO” (systolic blood pressure, multilobar chest radiography involvement, respiratory rate, tachycardia, confusion, and oxygenation).

A SMART-COP score of <3 points identified 84 (92.3%) of 91 patients who received IRVS. In comparison, a SMRT-CO score of <2 points identified 82 (90.1%) of the 91 patients. The sensitivities of all 4 tools (SMART-COP, SMRT-CO, PSI, and CURB-65) for predicting receipt of IRVS in the ACAPS cohort are shown in table 5. Receiver operating characteristic curves and AUCs for each tool are shown in figure 2. As shown in figure 3, an increasing SMART-COP score was associated with an increased rate of receipt of IRVS. An increasing score was also associated with higher mortality, and 42 (84%) of the 50 patients who died had SMART-COP scores <3 points. A SMART-COP score <3 points had a positive predictive value (PPV) of 22.2%, and each subsequent 1-point increase in the score raised the PPV by ∼10% (data not shown).

Table 5

Comparison of the accuracy of assessment tools in predicting the receipt of intensive respiratory or vasopressor support.

Figure 2

Area under the receiver operating characteristic curve (AUC) analysis for the 4 severity assessment tools. CURB-65, tool based on confusion, urea level, respiratory rate, blood pressure, and age <65 years; PSI, pneumonia severity index.

Figure 3

A, Need for intensive respiratory or vasopressor support (IRVS) by prediction tool based on systolic blood pressure, multilobar chest radiography involvement, albumin level, respiratory rate, tachycardia, confusion, oxygenation, and arterial pH (SMART-COP) score. B, The 30-day mortality by SMART-COP score.

Of the 53 patients who received IRVS who were admitted directly to the ICU from the emergency department, a SMART-COP score <3 points accurately identified 52 (98.1%). Of the 38 patients transferred from the emergency department to the general ward and later to the ICU, 32 (84.2%) had a SMART-COP score <3 points. Patients who received IRVS but had SMART-COP scores <3 all experienced significant clinical deterioration at least 24–48 h after hospital admission; all but 1 of these patients received noninvasive ventilation.

Validation of SMART-COP. Results of the external validation of SMART-COP are shown in table 6. Overall, AUC analysis indicated good discrimination for SMART-COP scores. There was no evidence of lack of fit in any database, indicating that the prediction probability of SMART-COP for IRVS appeared to be good. Nevertheless, there were some limitations in this validation analysis. Because results for respiratory rate and hypoxia were recorded as binary results in most of these databases (i.e., above or below the relevant cutoff of the PSI), it was generally difficult to assess accurately the lower cutoff levels suggested by SMART-COP and SMRT-CO for these variables for patients aged ⩽50 years [3, 21, 23, 24]. Similarly, serum albumin level and arterial pH were recorded infrequently in the validation cohorts [3, 23, 24], and we followed the standard of assuming that patients with missing data had normal values [3]. This approach is likely to have resulted in a lower calculated sensitivity for SMART-COP and may explain why SMRT-CO appeared to have similar sensitivity in those databases. Nevertheless, despite these methodological differences in the CAP validation cohort studies, SMART-COP appeared to be a sensitive and specific predictor of patients with CAP who are likely to require IRVS.

Table 6

Validation of SMART-COP and SMRT-CO in predicting the receipt of intensive respiratory or vasopressor support (IRVS) in 5 external databases.

Discussion

Current pneumonia severity assessment tools, such as PSI and CURB-65, aim to predict the likely 30-day mortality, but this outcome is heavily dependent on the patient's age and comorbid illnesses, so these tools may not necessarily predict the need for ICU admission or IRVS [8–12]. In fact, such features may be important to clinicians in determining whether a patient's case should be designated NFR and therefore not appropriate for aggressive medical management. A SMART-COP score of <3 points better identified the majority of patients who received IRVS than did PSI classes IV and V and CURB-65 group 3. SMART-COP was accurate both for patients who went directly to the ICU from the emergency department (sensitivity, 98%) and for those who were initially admitted to the general ward before their condition deteriorated (sensitivity, 84%). Increasing SMART-COP scores were associated with an increasing likelihood of requiring IRVS. Similarly, SMRT-CO, which does not require investigations beyond chest radiography and pulse oximetry, also proved to be an accurate and simple system for identifying patients at a higher risk of severe disease, although less so than SMART-COP. Accuracy of both tools was high for both the derivation and the validation cohorts. The age-adjusted cutoffs for respiratory rate and hypoxia used in these tools are particularly useful for the identification of younger, previously healthy patients with severe CAP. Such patients are better able to increase tidal volume, instead of just respiratory rate, and so may not achieve a rate of 30 breaths/min despite having severe CAP. In addition, their PaO₂ must drop further from baseline to reach <60 mm Hg, compared with that of many elderly patients with preexisting respiratory or cardiac comorbidities.

Although the PSI severity assessment tool is accurate in predicting 30-day mortality, it is cumbersome and therefore less attractive for widespread use [25–27]. In addition, a patient can be assigned to PSI class V on the basis of their age and comorbidities alone, even with clinically mild CAP [3]. Similar to previous studies in which 14%–37% of patients who required ICU admission were in PSI classes I–III [3, 8–10, 12], we found that 26% of patients in our study were in PSI classes I–III.

CURB-65 has the advantage of simplicity [4, 25, 26]. However, in our population, it had poor sensitivity for predicting 30-day mortality and the need for IRVS, and many patients were categorized incorrectly as low risk. Population differences may partly explain the discrepancy; we had a higher ICU admission rate (13% in ACAPS vs. 5% in the original CURB-65 study), and we included patients who are nursing home residents [4]. The discriminatory power of CURB-65 appears to be reduced by the use of cutoffs for diastolic blood pressure of ⩽60 mm Hg and for serum urea level of only 7 mmol/L (table 2). Thus, many elderly patients are classified as having severe disease (CURB-65 group 3) on the basis of their age and minimally abnormal readings of diastolic blood pressure and urea level.

Several previous studies have compared existing severity assessment systems, such as PSI and CURB-65, for their ability to predict ICU admission [11, 12, 28]. Capelastegui et al. [11] found similar AUC results for the 2 systems. However, CURB-65 group 3 identified only 33% of those admitted to the ICU and 39% of those who received mechanical ventilation [11]. Buising et al. [12] compared these and other systems and found that none of these tools were ideal for predicting ICU admission and that a modified version of the British Thoracic Society rule performed better, although it was equivalent to the PSI when only patients with confirmed CAP were assessed [12]. Both studies were limited by the small numbers of ICU patients (26 and 45 patients). In addition, each author group found that AUC results were similar for CURB-65 and PSI, because of the better sensitivity and inferior specificity of PSI [11, 12]. Because most clinicians consider a test with high sensitivity to be preferable in serious situations, such as for patients who require IRVS, such equivalence of AUC results may lack clinical relevance. More recently, España et al. [29] developed a new prediction tool that is designed to predict both IRVS and death. However, this tool includes the criteria of age <80 years, severe hypoxia (PaO₂ <54 mm Hg or PaO₂/FiO₂ <250) and tachypnea (respiratory rate, >30 breaths/min); thus, this tool is likely to have poor sensitivity for younger patients. Among ACAPS participants, this tool had a sensitivity of 37% and specificity of 90%.

Our study has some limitations. First, nearly all ACAPS patients were admitted to the hospital; thus, the study included smaller numbers of patients in PSI classes I and II. Second, although we recruited patients from 6 centers, the majority of patients were recruited from 3 large, urban teaching hospitals. Thus, we cannot be certain that similar findings would be noted in a more diverse patient population. Third, comparison with other previous studies is somewhat difficult because our key severity outcome measure was receipt of IRVS, rather than simple ICU admission. However, we believe that IRVS is a more robust end point because it avoids possible confounding associated with differences in ICU admission criteria. Finally, the external validation of SMART-COP and SMRT-CO was complicated by the absence of some data and the binary nature of some variables in these databases. Nevertheless, these databases represent the best available and have been used to develop and validate the PSI and other severity tools. On the basis of our analysis, SMART-COP and SMRT-CO appeared to be highly accurate among this very large, nonderivation cohort of >7000 patients.

In conclusion, SMART-COP is a new, relatively simple, 8-variable tool that appears to identify accurately patients with CAP who will require IRVS. Our findings suggest that SMART-COP is likely to be a useful advance for clinicians in the accurate prediction of disease severity among patients with CAP.

ACAPS Collaboration

The ACAPS Collaboration consists of Patrick Charles, Lindsay Grayson, Robert Pierce, Barrie Mayall, Paul Johnson, Michael Whitby, John Armstrong, Graeme Nimmo, Wendy Munckhof, David Looke, Luke Garske, Geoffrey Playford, Andrew Fuller, Robert Stirling, Denis Spelman, Tom Kotsimbos, Peter Holmes, Tony Korman, Philip Bardin, Grant Waterer, Keryn Christiansen, Christopher Heath, Alistair Wright, Michael Catton, Christopher Birch, Julian Druce, Norbert Ryan, Lou Irving, and David Hart.

Acknowledgments

We acknowledge the assistance of the research nurses (Barbara Johnson, Michelle Hooy, Kathy Bailey, Sue Colby, Claire Forsdyke, and Bernadette Dunlop), medical staff, laboratory workers, and the assistant statistician (Cathy Martin), all of whom greatly assisted with the performance of ACAPS.

Financial support. Untied grants from the 20ICC Research Fund and the Victorian Department of Human Services.

Potential conflicts of interest. All authors: no conflicts.

Presented in part: 46th Annual Interscience Conference on Antimicrobial Agents and Chemotherapy, San Francisco, California, 27–30 September 2006 (abstract L-1075); 47th Annual Interscience Conference on Antimicrobial Agents and Chemotherapy, Chicago, Illinois, 17–20 September 2007 (abstract L-1156a).

a

Members of the collaboration are listed at the end of the text.