Telephone triage of chest pain in out-of-hours primary care: external validation of a symptom-based prediction rule to rule out acute coronary syndromes

Abstract

Introduction

Telephone triage is pivotal for evaluating the urgency of patient care, and in the Netherlands, the Netherlands Triage Standard (NTS) demonstrates moderate discrimination for chest pain. To address this, the Safety First Prediction Rule (SFPR) was developed to improve the safety of ruling out acute coronary syndrome (ACS) during telephone triage.

Methods

We conducted an external validation of the SFPR using data from the TRACE study, a retrospective cohort study in out-of-hours primary care. We evaluated the diagnostic accuracy assessment for ACS, major adverse cardiovascular events (MACE), and major events within 6 weeks. Moreover, we compared its performance with that of the NTS algorithm.

Results

Among 1404 included patients (57.3% female, 6.8% ACS, 8.6% MACE), the SFPR demonstrated good discrimination for ACS (C-statistic: 0.79; 95%-CI: 0.75–0.83) and MACE (C-statistic: 0.79; 95%-CI: 0.0.76–0.82). Calibration was satisfactory, with overestimation observed in high-risk patients for ACS. The SFPR (risk threshold 2.5%) trended toward higher sensitivity (95.8% vs. 86.3%) and negative predictive value (99.3% vs. 97.6%) with a lower negative likelihood ratio (0.10 vs. 0.34) than the NTS algorithm.

Conclusion

The SFPR proved robust for risk stratification in patients with acute chest pain seeking out-of-hours primary care in the Netherlands. Further prospective validation and implementation are warranted to refine and establish the rule’s clinical utility.

chest pain, primary care, acute coronary syndrome, clinical prediction rule

Key messages

• The Safety First Prediction Rule (SFPR) performs robust for risk stratification of acute chest pain in Dutch OOH-PC.

• The SFPR outperformed the current triage standard for chest pain evaluation.

• Prospective validation and monitored implementation are warranted.

Introduction

Telephone triage is often the first step in assessing whether a patient requires urgent care. To help primary care triage assistants in this process, triage protocols have been developed [1]. In the Netherlands, the leading triage protocol in primary care is the “Netherlands Triage Standard” (NTS) [2]. This protocol was derived from the Manchester triage protocol and adapted to a Dutch primary care setting. Prior evaluation of the NTS triage protocol shows that it has moderate discrimination for chest pain, a common symptom that may among others be the result from acute coronary syndrome (ACS) [3]. Given the underperformance of this system, a new symptom-based prediction rule, the Safety First Prediction Rule (SFPR), was developed for chest pain, with the primary goal of safely ruling out ACS [4]. This model consists of seven predictors, and internal–external cross validation showed good discriminative ability for ACS within 30 days after telephone triage (adjusted C-statistic 0.77 (95%CI: 0.74–0.79)). In the current study, we aimed to perform external validation of the SFPR for ACS and other relevant outcomes in a sample of patients who contacted an out-of-hours primary care (OOH-PC) facility for acute chest pain in a different region of the Netherlands.

Methods

We reported our study in accordance with the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis (TRIPOD) statement and used the checklist for Prediction Model Validation [5]. The Medical Ethics Committee of the Amsterdam University Medical Center, reviewed and approved our study protocol (National Trial Register NL7581) [6].

Study design and participants

We assessed the diagnostic accuracy of the SFPR using data from the “TRiage of Acute Chest pain Evaluation in primary care (TRACE)” study [3, 7]. TRACE is an observational, retrospective cohort study of patients with chest pain who contacted OOH-PC between 1 January 2017 and 31 December 2017. For this study, all consecutive patients ≥18 years with acute chest pain were considered eligible for inclusion at their first contact with the OOH-PC facility in Alkmaar, the Netherlands. At the outset of the TRACE study, all eligible patients received information by mail and were provided the opportunity to opt-out from sharing data. Patients who expressed objection to participation and those patients who bypassed the triage system were not eligible for inclusion. We further excluded patients with missing data on any of the SFPR elements from the analyses.

In the Netherlands, OOH-PC facilities provide care to the patients of affiliated GPs when daytime practices are closed (i.e. evenings, nights, weekends, and holidays). Given that referral by a GP is required for secondary care in the Netherlands, patients with chest pain will generally first present to OOH-PC facilities, with the exception of cases where immediate hospitalization is indicated. The facility in Alkmaar provides care to approximately a quarter million patients and covers both urban and rural areas. Upon telephone contact with the OOH-PC facility, trained triage assistants gather information on patient and symptom characteristics to assess the patient’s urgency (using urgency codes U1-U5, in which U1 indicates the highest urgency) and determine the preferred course of action (e.g. ambulance activation, GP consultation, telephone advice). Here, triage assistants use standardized, symptom-based protocols from the NTS.

Safety First Prediction Rule

The SFPR consists of a multivariate model with seven predictors: (i) age, (ii) sex, (iii) acute onset of chest pain (<12 h), (iv) a pressing/heavy character, (v) radiation of pain, (vi) sweating, and (vii) calling at night (i.e. between 00.00 and 09.00). The model was developed using a multivariate regression analysis with candidate predictors derived from the NTS protocol, existing chest pain risk scores, and prior analyses by the research group [4]. Table 1 displays model variables, interaction terms, and applied coefficients verified by correspondence with the original authors [4].

Table 1.

Components of the SFPR.

Model variable	Original SFPR regression coefficients (SE)	Differencea	P value
Intercept	−16.246 (3.527)	7.068	0.013
Age	0.293 (0.081)	−0.182	0.008
Age'	−0.391 (0.125)	0.316	0.011
Age''	1.063 (0.395)	−1.062	0.017
Female sex	2.504 (5.512)	−32.890	0.035
Acute onset of chest pain (< 12 h)	0.290 (0.198)	0.315	0.254
Sweating	0.457 (0.178)	0.282	0.236
Radiation of chest pain	0.609 (0.176)	0.086	0.708
Pressing or heavy pain	0.747 (0.200)	−0.142	0.576
Calling at night between 00.00 and 9.00 h	0.504 (0.151)	−0.213	0.366
Age × female sex	−0.096 (0.126)	0.765	0.026
Age' × female sex	0.189 (0.195)	−1.173	0.014
Age'' × female sex	−0.556 (0.605)	3.552	0.010

^aDifference between the SFPR and TRACE coefficients when adding the SFPR linear predictor as an offset in the TRACE dataset for the primary outcome (acute coronary syndrome).

Knots for cubic spline functions placed at 25, 51, 69, and 88. SE, standard error; SFPR, Safety First prediction rule.

In the TRACE database, age, sex, and the time of contact are registered by default for each patient who contacts the OOH-PC facility. The other predictors consist of symptom characteristics that are part of the NTS protocol and are therefore readily available in our patient sample. Researchers coding the predictors from electronic health records were blinded for the outcome. Medical history items and symptom characteristics that were not explicitly mentioned in the triage registries were assumed absent and were coded as such.

Outcomes and reference standard

The primary outcome was ACS, defined as acute myocardial infarction (i.e. NSTEMI or STEMI) or unstable angina, as the underlying diagnosis for the index telephone contact. Secondary outcomes were major adverse cardiovascular events (MACE) and the occurrence of any major event up to 6 weeks after index contact. MACE is defined as the composite of all-cause mortality, acute myocardial infarction, and acute coronary interventions (i.e. percutaneous coronary intervention or coronary artery bypass graft surgery) [8]. We defined major events as all-cause mortality or the occurrence of any cardiovascular and noncardiovascular urgent condition that was linked to the initial complaint of chest pain if these required hospital admission or immediate in-hospital treatment (Supplementary Table S1) [3, 7].

Reference diagnoses were established using a delayed-type reference standard. In the Netherlands, each inhabitant is registered with a GP practice that holds a patient record file, including correspondence from OOH-PC facilities, emergency departments, and hospital specialists. Follow-up data, including final diagnoses, were collected from these patient record files at least 6 weeks after initial contact with the OOH-PC facility, in order to assess MACE and major events and to allow for any delayed correspondence relating to ACS to have arrived back at the participating patient’s GP. The researchers (A.M., R.E.H.) assessing the occurrence of ACS, MACE, or major events were not blinded to information from the triage report. An independent expert was available as a third assessor in case of disagreement.

Statistical analysis

We reported descriptives of categorical variables as number and percentage, and continuous and ordinal variables as median and interquartile range (IQR). We assessed differences using Pearson chi² test for categorical variables, and Student’s t-test (normally distributed) or Mann–Whitney U test (non-normally distributed) for continuous data. We presented baseline characteristics for the complete cases as well as for the cases excluded for missing data for ≥1 of the SFPR variables and provided the percentage missing for each of the SFPR variables.

For the external validation of the SFPR, we performed a complete case analysis of TRACE patients who had available data on the rule’s predictors and in whom a final diagnosis was established. We calculated risk as per the SFPR for each patient as 1/(1 + exp(−LP)), with LP being the linear predictor calculated as the sum of coefficients as provided by the authors of the original derivation paper times the variable values of the model [4]. We modeled age similarly to the derivation paper, using a restricted cubic spline function with knots at 25, 51, 69, and 88 years of age. The intercept and variable coefficients used in the analyses are shown in Table 1.

We assessed discrimination of the SFPR for the outcomes ACS, MACE, and major event separately by the C-statistic and 95% confidence interval (CI) using DeLong’s method [9]. In assessing calibration for each of the outcomes, we produced calibration plots and used the calibration slope of the linear predictor and its 95%-CI to assess the agreement between predicted and observed cases. In providing the calibration plot and slope, we corrected for optimism using 200 bootstrapping samples. We calculated the model’s diagnostic accuracy for predicting each of the outcomes using sensitivity, specificity, positive and negative predictive values (PPV and NPV), and positive and negative likelihood ratios (LR+ and LR−). We applied a threshold of 2.5%, corresponding to previous validation of the NTS for ACS, where 61.8% of presentations were assessed as high risk. In addition, we used the risk threshold corresponding to an NPV of 99% for ACS, which is generally accepted for the exclusion of ACS [10, 11]. In interpreting LR+ and LR−, an LR+>10 and LR− < 0.1 are generally considered to indicate that a test can rule a diagnosis in or out, respectively [12].We subsequently provided these same diagnostic accuracy parameters for high urgency as per original telephone triage (defined as NTS urgencies U1 or U2) for the outcomes of interest to facilitate a comparison between the SFPR and the current standard for telephone triage of chest pain.

To investigate possible differences in associations, we fitted the same model to our data, adding the Safety First linear predictor as an offset. We also provided the shrinkage factor for the overall model when validating the primary outcome in our dataset to further assess the external validity of the original SFPR and its coefficients. We did not proceed to recalibrate the model due to the limited sample size and risk of overfitting. Finally, we plotted the probabilities of the outcomes as a function of age for the men and women separately to visualize the significant interaction between age and sex. This was analogous to the analysis reported in the SFPR derivation paper, which showed a significant interaction between sex and age with significant differences among patients between the ages of 40 and 70.

Results

Between 1 January 1 2017 and 31 December 2017, 1803 eligible patients contacted the OOH-PC facility in Alkmaar regarding acute chest pain. A total of 399 (22.1%) patients with missing information on at least one of the model’s predictors were excluded from the analysis (Fig. 1). Patient characteristics and urgencies did not differ between these patients and those included in the final sample (Table 2). Among the 1404 included patients, the median age was 54 years (IQR: 37–71) and 57.3% were female. There were 95 ACS cases (5.5% among women and 8.5% among men, P = 0.024), composed of unstable angina (n = 19), NSTEMI (n = 41), STEMI (n = 24), and unspecified ACS (n = 11). The majority of included patients (61.4%) received a high urgency code (i.e. U1 or U2) following triage at the OOH-PC facility, including 87.4% of patients with ACS. Overall, the median estimated SFPR risk in our sample was 0.050 (IQR: 0.005–0.121). The risk was 0.032 (IQR: 0.003–0.081) in women and 0.085 (IQR: 0.011–0.172) in men.

Table 2.

Characteristics of TRACE patients included in the SFPR analyses.

Patient characteristics	Included (n = 1404)	Cases with missing SFPR data (n = 399)	Overall (n = 1803)
SFPR demographic variables
Age	54 (37–71)	52 (33–70)	54 (37–70)
Missing	0 (0.0)	0 (0.0)	0 (0.0)
Female sex	805 (57.3)	233 (58.4)	1038 (57.6)
Missing	0 (0.0)	0 (0.0)	0 (0.0)
SFPR symptoms
Chest pain duration < 12 h	852 (60.7)	23 (5.8)	875 (48.5)
Missing	0 (0.0)	349 (87.5)	349 (19.4)
Pressing/heavy chest pain	679 (48.4)	22 (5.5%)	701 (38.9%)
Missing	0 (0.0)	349 (87.5)	349 (19.4)
Radiation of pain to any location	618 (44.0)	23 (5.8)	641 (35.6)
Missing	0 (0.0)	376 (94.2)	376 (20.8)
Sweating	267 (19.0)	6 (1.5)	273 (15.1)
Missing	0 (0.0)	332 (83.2)	332 (18.4)
Calling at night (00:00–09:00)	347 (24.7)	18 (4.5%)	365 (20.2%)
Missing	0 (0.0)	358 (89.7)	358 (19.9)
Medical history and risk factors
History of coronary artery disease	256 (18.2)	40 (10.0)	296 (16.4)
Diabetes	172 (12.3)	27 (6.8)	199 (11.0)
Hypertension	432 (30.8)	58 (14.5)	490 (27.2)
Hypercholesterolaemia	235 (16.7)	16 (4.0)	251 (13.9)
Urgency
High urgency (U1/U2)	862 (61.4)	199 (49.9)	1061 (58.8)
Urgency U1	532 (37.9)	136 (34.1)	668 (37.0)
Urgency U2	330 (23.5)	63 (15.8)	393 (21.8)
Outcomes
ACS	95 (6.8)	4 (1.0)	99 (5.5)
Unstable angina	19 (1.4)	0 (0.0)	19 (1.1)
NSTEMI	41 (2.9)	3 (0.8)	44 (2.4)
STEMI	24 (1.7)	0 (0.0)	24 (1.3)
Not specified	11 (0.8)	1 (0.3)	12 (0.7)
MACE	121 (8.6)	7 (1.8)	128 (7.1)
Major event	223 (15.9)	15 (3.8)	238 (13.2)

This table shows the characteristics of TRACE patients included in the analyses (n = 1404). The middle column of the table illustrates the characteristics for the n = 399 patients that were excluded from the analyses due to missing data for the SFPR variables. Percentage of missing data per group is provided for each of the SFPR variables. The right column summarizes the characteristics among the total group of TRACE patients (n = 1803). Categorical variables are described using frequencies (%), age is described as median (IQR). ACS, acute coronary syndrome; MACE, major adverse cardiovascular event; NSTEMI, non-ST-elevation myocardial infarction; SFPR, Safety First prediction rule; STEMI, ST-elevation myocardial infarction; U1, Netherlands Triage Standard urgency level U1; U2, Netherlands Triage Standard urgency level U2.

Figure 1.

Flow of patients. The TRACE cohort comprised 1803 patients eligible for inclusion. Due to missing data on SFPR predictors or outcome measures, 399 patients were excluded from the final analyses. SFPR, Safety First Prediction Rule.

Safety First Prediction Rule validation for acute coronary syndrome

Assessment of the discriminatory ability showed a C-statistic of 0.79 (95%-CI: 0.75–0.83) for ACS. The calibration slope was 0.99 before and after bootstrap correction. The calibration plot indicated good overall calibration (Fig. 2).

Figure 2.

Calibration plot for the Safety First prediction rule for ACS. Brackets on the top horizontal axis indicate the proportion of patients with respective predicted risk (x axis).

Safety First Prediction Rule validation for major adverse cardiovascular events and major events

MACE occurred in 121 patients (7.1% among women and 10.7% among men, P = 0.017). Major events were identified in 223 patients (12.2% among women and 20.9% among men, P < 0.001), and were mostly of cardiovascular origin (76.7%), followed by respiratory (9.9%) and abdominal (7.2%) etiologies. The C-statistics (95%-CI) of the SFPR for MACE and major events were 0.79 (0.76–0.82) and 0.78 (0.75–0.81), respectively. Calibration of the SFPR for MACE showed good calibration for MACE in patients with low estimated risk but with a trend toward overestimation in the small group of patients with the highest predicted risks (Fig. 3, left). Calibration for major events was good in the majority of patients, with slight underestimation of risk in those with the highest predicted risks (Fig. 3, right). The calibration slope of the SFPR was 0.99 for both MACE and major events and was unchanged after correction for optimism.

Figure 3.

Calibration plots of the Safety First Prediction Rule for MACE and major events. Brackets on the top horizontal axis indicate the proportion of patients with respective predicted risk (x axis). MACE, major adverse cardiac events.

Diagnostic accuracy at different Safety First Prediction Rule risk thresholds and comparison with Netherlands Triage Standard

We calculated at which SFPR cutoff the NPV for ACS equaled 99%. This was established at a threshold of 3.3%, qualifying 57.3% of patients as high risk (Table 3). When applying the threshold for the other outcomes, the score performed sufficiently for ruling out MACE (NPV 99.0%), but not for major events (NPV 96.5%).

Table 3.

Diagnostic accuracy of SFPR and NTS for the outcomes of interest in the TRACE cohort.

	Risk threshold	% “high risk” at threshold	Sensitivity	Specificity	PPV	NPV	LR+	LR−
ACS	NTS U1/U2	61.5	86.3 (77.7–92.5)	40.4 (37.7–43.1)	9.5 (8.8–10.3)	97.6 (96.1–98.5)	1.45 (1.32–1.59)	0.34 (0.20–0.56)
	SFPR 0.025	61.5	95.8 (89.6–98.8)	41.3 (38.7–44.1)	10.6 (10.0–11.2)	99.3 (98.1–99.7)	1.63 (1.53–1.74)	0.10 (0.04–0.27)
	SFPR 0.033a	57.3	93.7 (86.8–97.7)	45.3 (42.6–48.0)	11.1 (10.4–11.8)	99.0 (97.9–99.5)	1.71 (1.59–1.84)	0.14 (0.06–0.30)
MACE	NTS U1/U2	61.5	85.1 (77.5–90.9)	40.8 (38.1–43.5)	12.0 (11.1–12.9)	96.7 (95.0–97.8)	1.44 (1.32–1.57)	0.36 (0.24–0.56)
	SFPR 0.025	61.5	96.7 (91.8–99.1)	42.2 (39.5–44.9)	13.6 (13.0–14.3)	99.3 (98.1–99.7)	1.67 (1.58–1.77)	0.08 (0.03–0.21)
	SFPR 0.033a	57.3	95.0 (89.5–98.2)	46.2 (43.5–49.0)	14.3 (13.5–15.1)	99.0 (97.8–99.5)	1.77 (1.66–1.89)	0.11 (0.05–0.23)
Major event	NTS U1/U2	61.5	82.5 (76.9–87.3)	42.6 (39.8–45.5)	21.4 (20.1–22.7)	92.8 (90.6–94.5)	1.44 (1.33–1.55)	0.41 (0.31–0.6)
	SFPR 0.025	61.5	93.3 (89.2–96.2)	44.9 (42.0–47.8)	24.2 (23.1–25.4)	97.3 (95.6–98.3)	1.69 (1.59–1.80)	0.15 (0.09–0.25)
	SFPR 0.033a	57.3	90.6 (86.0–94.1)	48.9 (46.1–51.8)	25.1 (23.8–26.4)	96.5 (94.8–97.7)	1.77 (1.65–1.90)	0.19 (0.13–0.29)

Sensitivity, specificity, PPV, and NPV are shown in percentages (%) with the 95%-CI between brackets.

ACS, acute coronary syndrome; LR+, positive likelihood ratio; LR−, negative likelihood ratio; MACE, major adverse cardiovascular event; NPV, negative predictive value; NTS U1/U2, Netherlands Triage Standard urgency level U1 or U2; PPV, positive predictive value; SFPR, Safety First prediction rule.

^aLowest cutoff where NPV for ACS best approached 99.0%.

When stratifying for NTS urgencies U1 or U2, 61.5% of the included patients were assessed as high risk. When comparing diagnostic accuracy parameters of the SFPR at 2.5% predicted risk with NTS’s U1/U2 urgency—each resulting in 61.5% of patients being assessed as having high risk—the SFPR trended toward higher sensitivity (95.8% vs 86.3%) and NPV (99.3% vs 97.6%) with lower LR- (0.10 vs 0.34) than the NTS triage urgencies (Table 3).

Supplementary Tables S2A–C illustrate the diagnostic accuracy of the SFPR and NTS for the outcomes of interest stratified by age, sex, and time of contact, respectively. Considerable differences were observed among patients stratified by age. Here, patients ≤54 years showed a sufficient NPV at a threshold of 3.3%, with only 22.3% of patients classified as high risk, whereas patients aged >54 years needed a SFPR threshold of 4.2% and 87.9% of patients were classified as high risk. When comparing women with men, optimal SFPR thresholds were set at 3.3% and 4.4%, respectively, classifying 49.2% of women and 66.6% of men as high risk. Stratifying for the time of contact did not result in considerable differences.

Additional analyses

Table 1 shows the results of adding the SFPR linear predictor as an offset in the TRACE dataset for the primary outcome (ACS). In TRACE, age and female sex, as well as their interaction variables, showed significantly different coefficients compared with the original SFPR coefficients. The shrinkage factor of the SFPR for ACS was 0.87 (95%-CI: 0.64–1.10), indicating that the overall deviations in model performance observed in the TRACE dataset were not statistically significant.

The interaction between age and sex for the probability of the outcomes of interest is illustrated in Fig. 4. For all outcomes (ACS, MACE, major events), risk increased with age for both women and men. However, only the risk of major events showed significant differences between women and men in the ages 55–80 years.

Figure 4.

Association between sex and age for the probability of ACS, MACE and major events in the TRACE cohort. ACS, acute coronary syndrome; MACE, major adverse cardiovascular events.

Discussion

In this study, we externally validated the SFPR for acute chest pain in OOH-PC. We found that the prediction rule had good discriminatory properties and was adequately calibrated. Moreover, it outperformed the current triage standard, with higher sensitivity and NPV, and lower LR-, indicating that the SFPR is better able to rule out chest pain using telephone triage in OOH-PC.

Strengths and limitations

External validation is pivotal prior to implementing a new risk score and/or decision support tool. In this study, we used a large, consecutive cohort of patients to conduct our analyses. Given the robust performance of the SFPR in an unrelated patient sample, we believe that the rule is generalizable to Dutch OOH-PC patients. While speculative, the rule might also be applicable in countries with similar health care systems, such as Scandinavian countries or the UK. We also extended our knowledge of the performance of the SFPR by evaluating the predictive value of the rule for other clinically relevant chest pain-related outcomes as well (i.e. MACE and major event).

The retrospective nature of our study constitutes its primary limitation. SFPR predictors were not systematically queried, leading to the exclusion of 22.1% of patients with missing data. Furthermore, our study used a delayed-type reference standard, which may introduce verification bias. However, we applied an extended follow-up period to accommodate any initially missed events, which is a common strategy in primary care-based studies. Another limitation is that our patient sample was too limited for recalibration and subsequent validation. Finally, triage in Dutch OOH-PC is based on key complaints (i.e. chest pain). This method is adopted in the SFPR, meaning that it assumes a key complaint of chest pain. Consequently, the rule is not tested in patients with atypical symptom presentations (e.g. without chest pain).

Comparison with existing literature

In the derivation paper of the SFPR, assessment of the final model and internal–external validation resulted in C-statistics of 0.79 (0.76–0.81) and 0.77 (0.74–0.79), respectively [4]. In our study, external validation confirmed the rule‘s discriminative abilities, with a C-statistic of 0.79 (0.75–0.83) for ACS. Calibration was satisfactory in the majority of patients, except for patients deemed at high risk (predicted probability >0.175), in whom absolute ACS risk was overestimated. This observation was found in both the original paper and our validation study. From a triage perspective, this may not be of clinical significance, as the risk threshold for high urgency and subsequent immediate action lies well below 0.175. Furthermore, in the original paper, the diagnostic test accuracy was calculated across a range of model-based risk thresholds, following the example of Wynants et al. [13]. We reproduced the threshold of 2.5% since this corresponds to the previous validation of the NTS for ACS, where 61.8% of presentations were assessed as high risk [3]. In addition, we used the risk threshold corresponding of an NPV of 99% for ACS and applied this for all outcomes. An NPV of 99% is generally accepted for the exclusion of ACS in both recommendations from the European Society of Cardiology and prior research among Dutch GPs [10, 11].

With our study, we expanded the assessment of the SFPR to other relevant chest pain-related outcomes by including MACE and major events as outcome measures. The rule showed robust results for MACE and major events, with C-statistics of 0.79 (0.76–0.82) and 0.78 (0.75–0.81), respectively. Calibration was acceptable for both.

Although we used a comparable sample of adult patients presenting to a Dutch OOH-PC facility with chest pain, differences between the original sample of Wouters et al. and our patients do exist. The original patient sample involved overall higher-risk patients, illustrated by an ACS prevalence of 11.5% (8.3% among women and 15.3% among men), compared to 6.8% (5.5% among women and 8.5% among men) in our patients [4]. Furthermore, in the derivation study, the interaction between age and sex was more outspoken. Here, the ACS risk increased with age for both sexes, with a noticeable peak in men around the age of 60 years, whereas women showed a more gradual increase in ACS risk. Significant differences in ACS probabilities were seen between women and men in the ages 45–70 years [4]. Among our patients, ACS risks increased with age, without significant differences between sexes. Differences in the interaction between age and sex might be caused by the amount of knots used for the cubic splines since too many knots might cause overfitting. However, reducing the amount of knots in the TRACE cohort (i.e. from four to three) did not affect our data.

For the derivation of the SFPR, the authors derived candidate predictors from current national triage guidelines (NTS) and previously developed decision rules, such as the Marburg Heart Score (MHS) and the INTERCHEST score [14, 15]. To assess the benefit of their final model, they compared its performance with that of the NTS (based on urgency code allocation, with high urgencies (U1 and U2) indicating a positive test). In their results, the NTS generated a C-statistic of 0.58 (4). Comparison of the Safety First model with the MHS and/or INTERCHEST score is lacking. A prior study by our own research group did compare the performance of the NTS to that of the MHS and INTERCHEST scores, when the latter two were applied as stand-alone triage tools. Here, the NTS resulted in a C-statistic of 0.68, which was outperformed by both the MHS and INTERCHEST scores, with C-statistics of 0.70 and 0.77, respectively [16].

The “calling at night” predictor in the SFPR is not part of the current NTS or existing clinical decision rules, but was derived from a previous study by Wouters et al. [17]. In this study, an increased risk of ACS was observed among patients who initiated contact with the OOH-PC facility during the night. Within our data, the “calling at night” predictor’s coefficient was slightly lower (offset −0.213). Therefore, we explored the effect of adjusting the time interval from 00:00–09:00 to 22:00–05:00 and recalculated risk prediction. However, this did not change the discriminative ability of the model (data not shown).

Implications for further research

In this retrospective, external validation of the SFPR, results are promising, and certainly when compared with the current triage standard. However, further prospective validation of the rule is warranted, especially if implementation outside the Netherlands is considered. Furthermore, the rule lacks prespecified risk thresholds, which impedes further evaluation.

Some previously developed clinical decision rules, such as the INTERCHEST score, include the GPs suspicion or “gut feeling” as a predictor for ACS [14]. In triage settings, triage assistants assess the patient’s urgency, often without the influence of GPs. Therefore, we might consider taking the triage assistant’s suspicion into account as an additional risk predictor in future research.

Conclusion

The SFPR was found to be a robust prediction tool for risk stratification of patients with acute chest pain who reached out to an OOH-PC facility in The Netherlands. The rule holds promise for improving future telephone triage but warrants further prospective validation and implementation.

Supplementary material

Supplementary material is available at Family Practice online.

Conflict of interest

None declared.

Funding

Funding for the TRACE study was provided by a grant from the Amsterdam Cardiovascular Sciences Research Institute (ACS-2018-TRACE-22254), a grant from ZonMw-HGOG (project number: 839150004), and internal funding from the department of general practice of the Amsterdam University Medical Centers, location AMC, the Netherlands.

Ethical approval

The authors assert that all procedures contributing to this work comply with the ethical standards of the relevant national and institutional guidelines on human experimentation and with the Helsinki Declaration of 1975, as revised in 2008. The TRACE study protocol was evaluated by the Medical Ethical Review Committee of the Amsterdam University Medical Centers, location AMC and was exempted full evaluation given the observational nature of the study. We reached out to data protection specialists and legal support how to best handle data protection and privacy of the individuals in the study. At the outset of the study, all patients received information by mail and were provided with the opportunity to opt-out from sharing data.

Data availability

Data are available on reasonable request by contacting the corresponding author.