Abstract

Most existing coronary risk assessment methods are based on baseline data only. The authors compared the predictive ability of coronary multivariable risk scores based on updated versus baseline risk factors and investigated the optimal frequency of updating. Data from 16 biennial examinations of 4,962 subjects from the original Framingham Heart Study (1948–1978) were used. The predictive ability of three multivariable risk scores was evaluated through 10-fold cross-validation. The baseline-only multivariable risk score was computed using baseline values of coronary risk factors applied to a Cox model estimated from baseline data. The two other approaches relied on updated risk factors and included them in the models estimated from, respectively, baseline and updated data. All analyses were stratified by sex and age. For 30, 14, and 10 years of follow-up, the predictive ability of the baseline-only multivariable risk score was substantially poorer than that of the models using updated risk factors. Between the two latter models, the one estimated from updated data ensured better prediction than the one estimated from baseline data for 30 years of follow-up among younger subjects only. The results suggest that coronary risk assessment can be improved by utilizing updated risk factors and that the optimal frequency of updating may vary across subpopulations.

Received for publication December 16, 2002; accepted for publication March 30, 2004.

Coronary heart disease is the leading cause of death in North America, making it important to accurately predict the risk of the disease for clinical, public health, and research purposes. Several methods for coronary heart disease prediction have been proposed (17). These methods allow calculation of an individual’s coronary multivariable risk score as a function of his/her values for selected established coronary heart disease risk factors. Apart from use in making clinical decisions, the multivariable risk score can be very useful in research. First, the multivariable risk score can serve as a confounder score (8) by providing a means of simultaneous adjustment for the effects of several coronary heart disease risk factors. This can be especially useful when studying coronary heart disease in smaller populations, such as patients with a rare disease (9, 10), where simultaneous modeling of multiple covariates is often not feasible because of the limited number of observed events (11). In addition, the background risk due to conventional coronary heart disease risk factors can act as an effect modifier for a postulated determinant of coronary heart disease, and then a single interaction term between the proposed determinant and the multivariable risk score can provide an efficient alternative to modeling multiple interactions with individual risk factors. Finally, the multivariable risk score can serve as a surrogate endpoint (12, 13) in studies that are not able to accrue a sufficient number of the clinical coronary heart disease events.

Most available risk-scoring methods rely on models built from initial risk factor values measured at study entry (17). This implies that these values do not change over time or that the initial measurements are the most relevant. However, at least some of the coronary risk factors show considerable intraindividual changes over time, so that the correlation between their initial and updated values gradually decreases with increasing follow-up (14, 15). If, in addition, the current risk depends mainly on recent risk factor values, the prognostic utility of a baseline value will decrease over time, because of regression dilution bias (16).

Therefore, it may be advantageous to incorporate more recent risk factor values into the multivariable model for coronary risk assessment. In fact, a recent coronary risk model, developed from the Framingham Study by D’Agostino et al. (17), did use updated risk factor data. In that study, the method of pooling repeated observations (18) was used to update risk factors at the beginning of each 2-year interval.

However, the assumption that updating coronary heart disease risk factors would improve their predictive utility has not been systematically evaluated. Although some studies support the use of updated measurements of such factors as smoking and serum cholesterol (1921), others suggest the opposite (2228). Moreover, each risk factor may have its own “optimal” time lag between the measurement and the outcome (27). This makes it difficult to determine a priori whether baseline or updated risk factor measurement is preferable for global coronary risk estimation and what is the optimal frequency of updating, calling for an empirical investigation.

One novel approach to investigating the use of baseline versus updated risk factors for coronary heart disease prediction was proposed by Cupples et al. (29), who applied pooled logistic regression to repeated measures from the Framingham Heart Study. Pooled logistic regression is a specific type of the pooling repeated observations technique, asymptotically equivalent (30) to the Cox regression with time-dependent covariates (31). Cupples et al. (29) compared parameter estimates for selected risk factors from the conventional baseline-only and pooled logistic regressions, but they did not compare the models’ overall predictive ability. In addition, the repeated observations were treated as independent (29), ignoring within-subject correlations, which could affect statistical inference (32, 33).

We attempted to systematically assess the potential advantages of using the multivariable risk score based on updated, instead of baseline, risk factors in coronary heart disease prediction and to establish the optimal frequency of updating.

MATERIALS AND METHODS

Study population

We used data from the first 30 years of follow-up, 1948–1978, in the Framingham Heart Study (34), with up to 15 postbaseline biennial examinations per subject. Subjects with prior coronary heart disease were excluded, as were those with missing values at both the baseline and second examinations for systolic blood pressure, daily number of cigarettes smoked, serum total cholesterol, body mass index, and glucose intolerance.

If age was reported at least once, missing age values at other examinations were imputed. Missing postbaseline values of the other risk factors were replaced with the most recent available value from the previous 4 years. Otherwise, the records with missing data were excluded.

Because of the different effects of some risk factors in men and women and in younger and older subjects (3538), all analyses were stratified by sex and baseline age, dichotomized at 28–49 years versus 50–62 years, except when indicated otherwise. The endpoint was the first occurrence of coronary heart disease, defined as myocardial infarction, angina pectoris, coronary insufficiency, or coronary heart disease death (39).

Statistical analysis

Predictors included age, current smoking, systolic blood pressure, body mass index, glucose intolerance, and serum total cholesterol. These were selected according to the recent Framingham scoring systems (3).

To evaluate the optimal time frame for measuring coronary heart disease risk factors for global coronary risk assessment, we used two approaches: “prognostic” and “lagged.” In the prognostic approach, only current and/or subsequent risk factor values were used for coronary heart disease prediction. In contrast, in the lagged approach, only risk factor values from the preceding 2 years (current) and earlier examinations were assessed for their relation to the development of coronary heart disease at a given examination. Thus, the prognostic method emulated the traditional approach to multivariable risk score development and application, with the pertinent question being if and how often the baseline risk factors should be updated. The lagged method was utilized as an alternative that could help to explain the results of the prognostic analyses by addressing the question of how much, if at all, the values of all the risk factors should be lagged to achieve the optimal association between the aggregate multivariable risk score and the observed outcomes.

Prognostic approach: estimation of multivariable models

Each of the four age/sex strata was analyzed separately. First, two different Cox proportional hazards multivariable models were estimated. In both models, the date of the first coronary heart disease event determined the time to event, and subjects were censored at the last examination or non-coronary heart disease death. The baseline-only Cox proportional hazards regression model relied on risk factor measurements at examination 1 (baseline), and this model was used for prediction of the first coronary heart disease event in the next 30 years. A similar approach was used in some coronary heart disease cohort studies, where only baseline measurements were available and used for prediction (e.g., in the Lipid Research Clinics study (40)). Use of this model in studies where risk factors are measured repeatedly implies that either the intraindividual fluctuations in risk factor levels during follow-up are considered negligible or the impact of such changes is very small, as postulated for older men, for example, by Benfante et al. (14). The second model relied on updated risk factors, represented with time-dependent covariates (31). Here, coronary heart disease events in the next 2 years were predicted using the risk factors from the most recent examination, consistent with the assumption that the current coronary heart disease risk is best determined by the recent risk factor values (17, 1921).

Next, we repeated similar analyses while limiting the follow-up to 14, 10, and 6 years. Accordingly, each subject could contribute up to two 14-year, three 10-year, or five 6-year observation periods, respectively, depending on how long the subject remained coronary heart disease free. For example, in the 6-year analysis, examinations 1, 4, 7, 10, and 13 provided “baseline” data for subsequent follow-up periods. At each baseline, subjects were regrouped into age strata depending on their current age.

Prognostic approach: model estimation and validation

To ensure unbiased comparison of the ability of the baseline-only Cox model and the Cox model with time-dependent covariates to predict outcomes in an independent sample from the same study population, we used a 10-fold cross-validation technique (41). Ten “training” and 10 corresponding “validation” subsets were generated by randomly dividing the original sample into 10 equally sized subsamples. Next, 10 separate models were fitted, each time using data from nine of the 10 subsamples (training subset) and leaving the 10th subsample out (validation subset). The data in the validation subsets were rearranged according to the pooling repeated observations technique (18), so that each subject contributed as many observations as there were 2-year intervals of his/her actual follow-up.

For each subject s in each validation subset, we first calculated three types of the multivariable risk score (MRS), having the general form MRS = Σβixis, where i = 1, …, and k indexes predictor variables. Each MRS was computed by first fitting a respective multivariable Cox regression model to the corresponding training subset and then multiplying the estimated coefficients (βi) by the subject’s risk factor values (xis). In MRS1, throughout the entire follow-up, each subject was assigned a constant MRS value (MRS1= Σβi0xi0s) computed using baseline risk factor values (x0) and parameter estimates from the baseline-only model (βi0). Calculation of MRS2 involved linking updated risk factor values (xits) with the estimates from the baseline-only model (βi0) (MRS2 = Σβi0xits). Thus, the MRS2 scores for an individual subject changed over time (t), reflecting within-subject variation in risk factors (xits). This simulated a hypothetical study in which risk factors were measured repeatedly over time, but the multivariable equation used to link risk factor values with the probability of outcome was derived from an external study that relied on baseline values only. Finally, MRS3 used risk factor values updated at each consecutive examination (xits) and the parameter estimates (βit) from the time-dependent Cox model that also relied on updated values (MRS3 = Σβitxits).

For each of the MRS types, in each of the validation subsets, the derived MRS was used as a single variable predicting coronary events in a generalized estimating equations generalization of the multivariable logistic regression model, with an order 1 autoregressive correlation structure of residuals (42) to account for the correlations between repeated measurements (33, 43). Thus, despite the conceptual differences at the model estimation stage between long-term prediction afforded by the baseline model and short-term prediction of the updated model, the use of multivariable risk scores at the validation stage allowed fair comparison of these methods. Indeed, in each MRS model, the outcome of an individual subject in each of the 2-year follow-up intervals was predicted using a model-specific aggregate score.

For each MRS type, the total cross-validated model deviance was computed by summing up the deviances from the 10 validation subsets. Deviance is calculated as twice the negative log-likelihood of the data and is a summary measure of discrepancies between predicted probabilities and actual outcomes of individual subjects. The validation was based only on the data not used to estimate the respective models, so that the cross-validated deviance approximated the predictive ability of the respective multivariable risk score in an independent sample from the same population (41). To compare the models’ ability to discriminate, in an independent sample, between subjects who will develop coronary heart disease during the relevant time period and those who will not, we calculated the R2 statistic (44) and the c-statistic (45), equivalent to the area under the receiver operating characteristic curve (46), by pooling results from the 10 validation subsets.

To assess statistical significance of the difference between the model-specific values of these statistics, we relied on bootstrap (47). For each time window, we generated 1,000 bootstrap resamples of the 10 original validation subsamples and pooled their results to obtain 1,000 model-specific bootstrap estimates of a given statistic. Next, for each pair of the MRS models, we calculated 1,000 differences between the corresponding statistics. Finally, the lower and upper boundaries of the bootstrap-based 95 percent confidence intervals for the difference between the two models were estimated as the 2.5th and 97.5th percentiles of the observed distribution of the 1,000 differences. A given difference was considered statistically significant at the 0.05 level (with two-tailed critical region) if the bootstrap 95 percent confidence interval excluded zero.

Lagged approach: estimation and validation of multivariable models

We compared the predictive ability of the multivariable risk score based on risk factors measured 2 (most recent), 6, 10, and 20 years prior to outcome ascertainment. For example, with a 20-year lag, the outcomes observed at examinations 14 and 15 are related to values observed at examinations 3 and 4, respectively. To enable comparisons with a 20-year lag, only surviving subjects who were coronary heart disease free until at least examination 11 were used in these analyses. Accordingly, the cutoff of 69 years was used to stratify by age. Starting from examination 11, each subject could thus contribute up to five assessments of his/her coronary heart disease status. Four separate, pooled, logistic generalized estimating equations regression models were developed, each using a different lag. When calculating the multivariable risk score, we used the risk factor values observed at the same time as the values used in model estimation. The four models were compared using the same 10-fold cross-validation procedure with bootstrap-based inference as for the prognostic approach.

RESULTS

Among the 5,209 subjects available at the baseline examination, 82 (1.6 percent) were excluded because of preexisting coronary heart disease. Those with missing baseline and examination 2 values for body mass index (n = 8; 0.2 percent), cigarette smoking (n = 35; 0.7 percent), and serum total cholesterol (n = 122; 2.4 percent) were also excluded. This left 4,962 subjects for the analyses, of whom 1,326 (26.7 percent) experienced a coronary event during the 30-year follow-up. Across a total of 55,527 examinations, 8 percent of systolic blood pressure values were missing and had to be imputed from earlier examinations. The percentages that were imputed for the other variables were 22 percent (serum cholesterol), 17 percent (glucose intolerance), 32 percent (current smoking), and 8 percent (body mass index). Table 1 summarizes the distributions of coronary heart disease events, follow-up duration, and baseline risk factors for the four age/sex strata.

Table 2 compares the predictive ability of the three prognostic MRS models over the 30-year follow-up period, according to the cross-validated values of deviance, the c-statistic (45), and maximum-rescaled R2 (44). For each stratum and each of the three performance indicators, the next column shows the value of the MRS3 model, which used updated risk factor values for both model estimation and score calculation. The three last columns show the mean of the 1,000 bootstrap resample-based differences in the corresponding statistics between a pair of models, together with the 95 percent bootstrap-based confidence interval. The asterisk indicates that a given difference is statistically significant at the 0.05 level.

MRS3 and MRS2, which both used updated risk factor values, predict outcomes much better than does baseline-only MRS1, with statistically significant differences in almost all comparisons (table 2). MRS2 and MRS3 perform quite similarly except for younger subjects, where the MRS3 yields lower deviances (table 2). While MRS2 and MRS3 also yielded significantly higher values of the c-statistic and R2 than did MRS1, the differences were less impressive than for the deviances, because these two measures of discriminatory ability are generally less sensitive to differences in predictive power of alternative models than are the deviance-based measures (48).

Tables 3 and 4 compare alternative models with follow-up limited to 14 and 10 years, respectively. Both MRS3 and MRS2 are definitely superior to MRS1 in all four strata, with statistically significant differences in all cases except c-statistics and R2 for younger women, while the performances of MRS3 and MRS2 are very similar.

Different results are obtained when the follow-up is limited to 6 years (table 5). MRS3 and MRS2 are still superior to MRS1 among the older subjects in both sexes, but among the younger subjects the three methods perform similarly.

Table 6 summarizes the results of the lagged approach, with different lag durations. For younger men, current risk factor values yield statistically significantly higher R2 and c-statistics than do the values observed 10 or 20 years earlier. For younger women, current risk factor values perform significantly better than do the values lagged by 20 years only. For older men, the current values yielded better results than did any of the lagged values, but the differences were not significant. In contrast, for older women, 6-year lagging improved significantly the R2 and c-statistics, compared with current values. In most of these cases, similar differences were observed for the corresponding deviances, but they never reached statistical significance.

When logistic regression, instead of the Cox model, was used for estimating the models in the training subsets, all the results were similar (data not shown).

To provide more insight regarding the differences between risk assessments based on baseline (MRS1) and updated (MRS3) risk factor values, we include figures 1 and 2, which present detailed data for a few randomly selected younger men. Each curve in these figures represents the changes over time in the 2-year probability of a coronary heart disease event, estimated by MRS3, for a particular subject. Figure 1 shows three subjects who did not develop coronary heart disease during the follow-up, and figure 2 shows three subjects who did. Although in both groups there is a gradual increase in coronary heart disease risk over time, the changes are not strictly monotonic and show between- as well as within-subject variability, reflecting different individual trajectories and their fluctuations over time. The increasing trend reflects the effect of aging on coronary heart disease risk, on which the effects of changes in the other risk factors are superimposed, leading to increases or decreases in the estimated risk.

For example, for subject A in figure 1, the coronary heart disease risk increases in the initial 2 years, then drops between years 2 and 4, mainly due to simultaneous declines in systolic blood pressure (from 133 to 112 mmHg), serum total cholesterol (from 326 to 255 mg/dl), and body mass index (from 29.4 to 28.2 kg/m2), but later increases steadily until year 16. Between years 16 and 18, the risk for subject A decreases considerably as a result of large decreases in systolic blood pressure (from 121 to 109 mmHg) and, especially, of cholesterol (from 326 to 266 mg/dl) and body mass index (from 28.1 to 26.4 kg/m2) and then remains relatively stable between years 18 and 22. However, a steep increase occurs at year 24 when the subject develops glucose intolerance. Finally, at year 26, the subject appears to reduce his risk again by decreasing the cholesterol level from 282 to 232 mg/dl. In contrast, subjects B and C show relatively stable, gradually increasing patterns in their predicted coronary heart disease risk (figure 1).

Figure 2 shows that the three subjects who ultimately developed coronary heart disease have, on average, less stable patterns in the estimated risk than the coronary heart disease-free subjects have in figure 1. For example, for subject D, the risk declines between years 2 and 6 because of decreasing systolic blood pressure (from 131 to 114 mmHg) and cholesterol (from 355 to 320 mg/dl) and then resumes a rather steady increase until year 24. Between years 24 and 26, the risk increases steeply, reflecting simultaneous increases in cholesterol (from 259 to 299 mg/dl) and body mass index (from 21.1 to 23.7 kg/m2). In the following 2 years, subject D develops glucose intolerance, but the overall risk decreases slightly as he considerably reduces cholesterol (from 299 to 257 mg/dl) and systolic blood pressure (from 154 to 137 mmHg). Even so, he experiences a coronary heart disease event in the next 2 years. Interestingly, subject F also shows an unstable up-and-down pattern of changes in risk just before the event (figure 2).

Overall, figures 1 and 2 show that the updated scores of MRS3 may reflect some important within-individual changes in risk over time. In contrast, the MRS1, which uses baseline values only, is by definition unable to capture such changes.

DISCUSSION

We have compared the abilities of three alternative multivariable risk score systems to predict development of coronary heart disease in different age- and sex-specific subgroups of the Framingham population. Our results indicate that assessment of short-term coronary risk can be improved by using updated risk factor values for calculation of the multivariable risk score regardless of whether the underlying equation was estimated from the updated or baseline risk factor data. In our analyses, most of these improvements were statistically significant, and their clinical relevance was reflected by the fact that, for example, R2 and deviance often improved by more than 30 percent and 15 percent, respectively, relative to the baseline-only MRS1. Furthermore, the differences in deviances often exceeded 100, indicating a very important improvement in prediction (41). However, the optimal frequency and utility of updating may vary across populations.

We have investigated the issue of optimal frequency for updating using two approaches. Despite only partial overlap between the younger subgroups in the prognostic (25–50 years at baseline) and lagged approaches (49–69 years at the time of coronary heart disease outcome), the results are generally consistent. Both approaches suggest that, while frequent (every 2 years) updating of risk factors improves prediction when compared with rare updating (every 10 or more years) in all the subgroups, updating every 2 years may not be preferable to updating every 6 years.

Our results corroborate previous findings that changes in cholesterol, blood pressure, and smoking lead to changes in coronary risk within the next 2–5 years (1921) and that these risk factors show considerable intraindividual fluctuations over time (14, 15, 4951). However, in contrast to previous studies, our analyses allow us to directly quantify the benefits derived from the systematic updating of the global multivariable risk score. Still, it is not immediately clear from our analyses which risk factor’s being updated benefited the models’ predictive ability the most, as the observed differences between the two multivariable Cox models (baseline only vs. time dependent) represent the net result of several jointly acting effects.

Our study has some limitations. Because we used data from the original Framingham Heart Study (1948–1978), the temporal generalizability of our prediction models might be reduced. However, previous validation studies have established the reasonably high accuracy of the Framingham equations for more recent populations (52), and thus we expect our methodological conclusions to apply to contemporary populations as well. Second, many risk factor values had to be imputed. Although carrying the last observation forward makes the comparisons more conservative, we were still able to demonstrate statistically significant improvements due to using updated rather than baseline risk factor values. However, that does not mean that the resulting coronary heart disease prediction was optimal. It is quite possible that each risk factor has its own lag, after which its impact on coronary heart disease risk is the strongest, and accounting for such risk factor-specific lags might lead to further improvements. Yet, we intentionally refrained from searching for risk factor-specific lags, as both the reliability of potential findings and the practicability of the resulting models would be questionable. Indeed, in both research settings and clinical practice, it is much more practical to consider the same time frame for all risk factors. In addition, there could be lag-specific functional forms of the relation between a coronary risk factor and coronary heart disease risk, as suggested by Emond and Zareba (53). These complex issues need to be investigated in future studies.

In spite of these limitations, our findings have practical implications. First of all, they show that, in clinical practice, an individual patient’s short-term prognosis could be improved by relying on the multivariable risk score that uses recent risk factor values. Second, the updated multivariable risk score provides a parsimonious means of simultaneous adjustment (8) for several coronary heart disease risk factors, which can be advantageous in coronary heart disease studies of smaller populations (10). Finally, our results suggest that the multivariable risk score should be updated frequently if it is used as a surrogate endpoint (12, 13) instead of the clinical coronary heart disease outcomes.

In conclusion, our analyses suggest that further refinement of the coronary heart disease prediction models, by accounting for additional information provided by updated risk factor values, may offer material benefits. However, further work is necessary to develop the optimal coronary heart disease risk scoring systems. Future research should consider incorporation of the emerging risk factors (54), as well as a more comprehensive assessment of the role of baseline values of particular risk factors and their subsequent changes. We believe that our results may encourage such further developments, which should ultimately result in improved prediction and prevention of coronary heart disease.

ACKNOWLEDGMENTS

This research was supported in part by the Medical Research Council of Canada (grant MA-13738) and the Canadian Institutes for Health Research (grant 82349). I. K. was supported by a Canadian Cardiovascular Outcomes Research Team PhD fellowship and by a fellowship from the Canadian Institutes for Health Research. M. A. is a James McGill Professor. L. P. is an investigator of the Canadian Institutes for Health Research and a Dawson Scholar at McGill University. G. B. is a Canadian Institutes for Health Research New Emerging Team New Investigator for Circulatory and Respiratory Diseases. The Framingham Heart Study is conducted and supported by the National Heart, Lung, and Blood Institute in collaboration with the Framingham Heart Study investigators.

This manuscript was not prepared in collaboration with investigators of the Framingham Heart Study and does not necessarily reflect the opinions or views of the Framingham Heart Study or the National Heart, Lung, and Blood Institute.

Correspondence to Dr. Michal Abrahamowicz, Division of Clinical Epidemiology, Montreal General Hospital, 1650 Cedar Avenue, Montreal, Quebec, Canada H3G 1A4 (e-mail: michal@michal.ri.mgh.mcgill.ca).

FIGURE 1. Plot of estimated 2-year coronary heart disease (CHD) risk over time in three selected subjects who did not develop CHD: subject A (solid line), subject B (dotted line), and subject C (dashed line), Framingham Heart Study, 1948–1978. Men were aged 28–49 years at baseline.

FIGURE 1. Plot of estimated 2-year coronary heart disease (CHD) risk over time in three selected subjects who did not develop CHD: subject A (solid line), subject B (dotted line), and subject C (dashed line), Framingham Heart Study, 1948–1978. Men were aged 28–49 years at baseline.

FIGURE 2. Plot of estimated 2-year coronary heart disease (CHD) risk over time in three selected subjects who developed CHD: subject D (solid line), subject E (dotted line), and subject F (dashed line), Framingham Heart Study, 1948–1978. Men were aged 28–49 years at baseline.

FIGURE 2. Plot of estimated 2-year coronary heart disease (CHD) risk over time in three selected subjects who developed CHD: subject D (solid line), subject E (dotted line), and subject F (dashed line), Framingham Heart Study, 1948–1978. Men were aged 28–49 years at baseline.

TABLE 1.

Follow-up and distributions of risk factors at baseline, by age and sex strata, Framingham Heart Study, 1948–1978

Risk factor Men   Women  
Age category, years  28–49 50–62  28–49 50–62 
No. of subjects 1,559 665  1,917 821 
Subjects with CHD* events (no. (%))† 479 (30.7) 282 (42.4)  297 (15.5) 268 (32.6) 
Follow-up, years (mean (SD*)) 26.4 (5.0) 23.0 (7.6)  28.6 (2.4) 25.6 (6.0) 
Total person-years of follow-up 35,678 11,614  47,192 16,624 
Age, years (mean (SD)) 39.3 (5.5) 54.7 (3.3)  39.4 (5.4) 54.6 (3.2) 
Total serum cholesterol, mg/dl (mean (SD)) 221.6 (43.4) 226.5 (42.2)  214.8 (42.8) 251.3 (46.8) 
Systolic BP,* mmHg (mean (SD)) 133.4 (16.5) 142.4 (22.6)  128.2 (18.0) 150.2 (46.8) 
Body mass index, kg/m2 (mean (SD)) 25.7 (3.5) 26.0 (3.4)  24.6 (4.4) 27.1 (4.9) 
Current smoking (no. (%)) 1,069 (68.6) 367 (55.2)  921 (48.0) 210 (25.5) 
Cigarettes per day among current smokers (mean (SD)) 22.7 (11.0) 19.9 (10.8)  13.5 (8.6) 11.7 (8.9) 
Glucose intolerance (no. (%)) 72 (4.6) 49 (7.4)  47 (2.5) 50 (6.1) 
Risk factor Men   Women  
Age category, years  28–49 50–62  28–49 50–62 
No. of subjects 1,559 665  1,917 821 
Subjects with CHD* events (no. (%))† 479 (30.7) 282 (42.4)  297 (15.5) 268 (32.6) 
Follow-up, years (mean (SD*)) 26.4 (5.0) 23.0 (7.6)  28.6 (2.4) 25.6 (6.0) 
Total person-years of follow-up 35,678 11,614  47,192 16,624 
Age, years (mean (SD)) 39.3 (5.5) 54.7 (3.3)  39.4 (5.4) 54.6 (3.2) 
Total serum cholesterol, mg/dl (mean (SD)) 221.6 (43.4) 226.5 (42.2)  214.8 (42.8) 251.3 (46.8) 
Systolic BP,* mmHg (mean (SD)) 133.4 (16.5) 142.4 (22.6)  128.2 (18.0) 150.2 (46.8) 
Body mass index, kg/m2 (mean (SD)) 25.7 (3.5) 26.0 (3.4)  24.6 (4.4) 27.1 (4.9) 
Current smoking (no. (%)) 1,069 (68.6) 367 (55.2)  921 (48.0) 210 (25.5) 
Cigarettes per day among current smokers (mean (SD)) 22.7 (11.0) 19.9 (10.8)  13.5 (8.6) 11.7 (8.9) 
Glucose intolerance (no. (%)) 72 (4.6) 49 (7.4)  47 (2.5) 50 (6.1) 

* CHD, coronary heart disease; SD, standard deviation; BP, blood pressure.

† Each subject was counted only once, even if he/she might have had several CHD events during the follow-up.

TABLE 2.

Comparison of cross-validated predictive ability between coronary heart disease risk prediction models based on three types of multivariable risk scores,† by age and sex strata, Framingham Heart Study, 1948–1978

Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 2,461 –331 (–381, –280)* –12 (–24, –2)* –319 (–368, –269)* 
Male 50–62 Deviance 1,270 –223 (–261, –181)* –1 (–14, 10) –223 (–259, –182)* 
Female 28–49 Deviance 1,686 –318 (–368, –264)* –24 (–44, –6)* –295 (–344, –241)* 
Female 50–62 Deviance 1,337 –199 (–232, –158)* –2 (–7, 3) –197 (–231, –157)* 
Male 28–49 c-statistic 0.72 0.02 (0.00, 0.04)* –0.01 (–0.01, 0.00) 0.02 (0.00, 0.04)* 
Male 50–62 c-statistic 0.63 0.01 (0.00, 0.03)* 0.01 (0.01, 0.02)* 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.80 0.05 (0.02, 0.08)* 0.01 (–0.01, 0.02) 0.04 (0.01, 0.08)* 
Female 50–62 c-statistic 0.68 0.09 (0.06, 0.11)* 0.03 (0.02, 0.04)* 0.06 (0.03, 0.08)* 
Male 28–49 R2 statistic 0.062 0.009 (–0.001, 0.020) –0.003 (–0.007, 0.000) 0.013 (0.003, 0.023)* 
Male 50–62 R2 statistic 0.027 0.007 (0.002, 0.011)* 0.003 (0.002, 0.005)* 0.003 (–0.001, 0.008) 
Female 28–49 R2 statistic 0.098 0.034 (0.015, 0.053)* 0.006 (–0.002, 0.014) 0.027 (0.010, 0.045)* 
Female 50–62 R2 statistic 0.044 0.027 (0.018, 0.037)* 0.011 (0.007, 0.015)* 0.016 (0.009, 0.023)* 
Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 2,461 –331 (–381, –280)* –12 (–24, –2)* –319 (–368, –269)* 
Male 50–62 Deviance 1,270 –223 (–261, –181)* –1 (–14, 10) –223 (–259, –182)* 
Female 28–49 Deviance 1,686 –318 (–368, –264)* –24 (–44, –6)* –295 (–344, –241)* 
Female 50–62 Deviance 1,337 –199 (–232, –158)* –2 (–7, 3) –197 (–231, –157)* 
Male 28–49 c-statistic 0.72 0.02 (0.00, 0.04)* –0.01 (–0.01, 0.00) 0.02 (0.00, 0.04)* 
Male 50–62 c-statistic 0.63 0.01 (0.00, 0.03)* 0.01 (0.01, 0.02)* 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.80 0.05 (0.02, 0.08)* 0.01 (–0.01, 0.02) 0.04 (0.01, 0.08)* 
Female 50–62 c-statistic 0.68 0.09 (0.06, 0.11)* 0.03 (0.02, 0.04)* 0.06 (0.03, 0.08)* 
Male 28–49 R2 statistic 0.062 0.009 (–0.001, 0.020) –0.003 (–0.007, 0.000) 0.013 (0.003, 0.023)* 
Male 50–62 R2 statistic 0.027 0.007 (0.002, 0.011)* 0.003 (0.002, 0.005)* 0.003 (–0.001, 0.008) 
Female 28–49 R2 statistic 0.098 0.034 (0.015, 0.053)* 0.006 (–0.002, 0.014) 0.027 (0.010, 0.045)* 
Female 50–62 R2 statistic 0.044 0.027 (0.018, 0.037)* 0.011 (0.007, 0.015)* 0.016 (0.009, 0.023)* 

* p < 0.05 (differences statistically significant with a two-tailed test).

† Multivariable risk score (MRS): MRS1 is based on regression coefficients from baseline-only Cox regression model and baseline risk factor values; MRS2 is based on regression coefficients from baseline-only Cox regression model and updated risk factor values; MRS3 is based on regression coefficients from time-dependent Cox regression model and updated risk factor values.

‡ Deviance, twice negative log-likelihood, by which lower values indicate better predictive ability; c-statistic, higher values indicate better predictive ability; R2 statistic,maximum-rescaled R2, by which higher values indicate better predictive ability.

§ This column shows the value of the given statistic (measure of predictive ability) for MRS3.

¶ Each of columns 5–7 compares two MRS models, with respect to the statistics in the third column, by showing the mean difference between the two models, across 1,000 bootstrap resamples, and the corresponding 95% bootstrap-based confidence interval.

TABLE 3.

Comparison of cross-validated predictive ability between coronary heart disease risk prediction models based on three types of multivariable risk scores,† by age and sex strata with 14-year follow-up, Framingham Heart Study, 1948–1978

Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 1,369 –105 (–133, –75)* –1 (–7, 4) –104 (–131, –75)* 
Male 50–62 Deviance 2,313 –201 (–233, –166)* –3 (–11, 4) –198 (–231, –164)* 
Female 28–49 Deviance 647 –47 (–75, –16)* 1 (–5, 8) –48 (–75, –17)* 
Female 50–62 Deviance 2,326 –206 (–243, –171)* –2 (–9, 5) –205 (–241, –170)* 
Male 28–49 c-statistic 0.72 0.02 (0.01, 0.04)* 0.00 (–0.01, 0.00) 0.03 (0.01, 0.05)* 
Male 50–62 c-statistic 0.63 0.03 (0.02, 0.05)* 0.01 (0.00, 0.02)* 0.02 (0.01, 0.04)* 
Female 28–49 c-statistic 0.78 0.01 (–0.02, 0.04) 0.00 (–0.01, 0.02) 0.01 (–0.02, 0.04) 
Female 50–62 c-statistic 0.66 0.03 (0.01, 0.04)* 0.00 (–0.01, 0.01) 0.03 (0.02, 0.05)* 
Male 28–49 R2 statistic 0.064 0.017 (0.008, 0.027)* –0.002 (–0.005, 0.000) 0.020 (0.010, 0.030)* 
Male 50–62 R2 statistic 0.026 0.012 (0.007, 0.018)* 0.004 (0.002, 0.006)* 0.008 (0.003, 0.013)* 
Female 28–49 R2 statistic 0.083 0.001 (–0.016, 0.018) –0.009 (–0.018, 0.000)* 0.010 (–0.009, 0.027) 
Female 50–62 R2 statistic 0.038 0.011 (0.004, 0.018)* 0.000 (–0.003, 0.003) 0.011 (0.005, 0.018)* 
Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 1,369 –105 (–133, –75)* –1 (–7, 4) –104 (–131, –75)* 
Male 50–62 Deviance 2,313 –201 (–233, –166)* –3 (–11, 4) –198 (–231, –164)* 
Female 28–49 Deviance 647 –47 (–75, –16)* 1 (–5, 8) –48 (–75, –17)* 
Female 50–62 Deviance 2,326 –206 (–243, –171)* –2 (–9, 5) –205 (–241, –170)* 
Male 28–49 c-statistic 0.72 0.02 (0.01, 0.04)* 0.00 (–0.01, 0.00) 0.03 (0.01, 0.05)* 
Male 50–62 c-statistic 0.63 0.03 (0.02, 0.05)* 0.01 (0.00, 0.02)* 0.02 (0.01, 0.04)* 
Female 28–49 c-statistic 0.78 0.01 (–0.02, 0.04) 0.00 (–0.01, 0.02) 0.01 (–0.02, 0.04) 
Female 50–62 c-statistic 0.66 0.03 (0.01, 0.04)* 0.00 (–0.01, 0.01) 0.03 (0.02, 0.05)* 
Male 28–49 R2 statistic 0.064 0.017 (0.008, 0.027)* –0.002 (–0.005, 0.000) 0.020 (0.010, 0.030)* 
Male 50–62 R2 statistic 0.026 0.012 (0.007, 0.018)* 0.004 (0.002, 0.006)* 0.008 (0.003, 0.013)* 
Female 28–49 R2 statistic 0.083 0.001 (–0.016, 0.018) –0.009 (–0.018, 0.000)* 0.010 (–0.009, 0.027) 
Female 50–62 R2 statistic 0.038 0.011 (0.004, 0.018)* 0.000 (–0.003, 0.003) 0.011 (0.005, 0.018)* 

* p < 0.05 (differences statistically significant with a two-tailed test).

† Multivariable risk score (MRS): MRS1 is based on regression coefficients from baseline-only Cox regression model and baseline risk factor values; MRS2 is based on regression coefficients from baseline-only Cox regression model and updated risk factor values; MRS3 is based on regression coefficients from time-dependent Cox regression model and updated risk factor values.

‡ Deviance, twice negative log-likelihood, by which lower values indicate better predictive ability; c-statistic, higher values indicate better predictive ability; R2 statistic,maximum-rescaled R2, by which higher values indicate better predictive ability.

§ This column shows the value of the given statistic (measure of predictive ability) for MRS3.

¶ Each of columns 5–7 compares two MRS models, with respect to the statistics in the third column, by showing the mean difference between the two models, across 1,000 bootstrap resamples, and the corresponding 95% bootstrap-based confidence interval.

TABLE 4.

Comparison of cross-validated predictive ability between coronary heart disease risk prediction models based on three types of multivariable risk scores,† by age and sex strata with 10-year follow-up, Framingham Heart Study, 1948–1978

Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 991 –51 (–80, –29)* 1 (–5, 7) –55 (–81, –30)* 
Male 50–62 Deviance 2,667 –187 (–221, –155)* –3 (–8, 2) –184 (–215, –152)* 
Female 28–49 Deviance 448 –32 (–49, –14)* –1 (–10, 6) –31 (–48, –14)* 
Female 50–62 Deviance 2,468 –203 (–235, –166)* –2 (–7, 3) –201 (–233, –163)* 
Male 28–49 c-statistic 0.72 0.02 (0.00, 0.04) –0.01 (–0.01, 0.00) 0.02 (0.00, 0.04) 
Male 50–62 c-statistic 0.63 0.01 (0.00, 0.03) 0.01 (0.01, 0.02)* 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.80 0.05 (0.02, 0.08)* 0.01 (–0.01, 0.02) 0.04 (0.01, 0.08)* 
Female 50–62 c-statistic 0.68 0.04 (0.02, 0.05)* 0.00 (0.00, 0.01) 0.03 (0.02, 0.05)* 
Male 28–49 R2 statistic 0.062 0.009 (–0.001, 0.020) –0.003 (–0.007, 0.000) 0.013 (0.003, 0.023)* 
Male 50–62 R2 statistic 0.027 0.007 (0.002, 0.011)* 0.003 (0.002, 0.005)* 0.003 (–0.001, 0.008) 
Female 28–49 R2 statistic 0.098 0.034 (0.015, 0.053)* 0.006 (–0.002, 0.014) 0.027 (0.001, 0.045)* 
Female 50–62 R2 statistic 0.044 0.017 (0.011, 0.023)* 0.001(–0.001, 0.004) 0.016 (0.010, 0.022)* 
Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 991 –51 (–80, –29)* 1 (–5, 7) –55 (–81, –30)* 
Male 50–62 Deviance 2,667 –187 (–221, –155)* –3 (–8, 2) –184 (–215, –152)* 
Female 28–49 Deviance 448 –32 (–49, –14)* –1 (–10, 6) –31 (–48, –14)* 
Female 50–62 Deviance 2,468 –203 (–235, –166)* –2 (–7, 3) –201 (–233, –163)* 
Male 28–49 c-statistic 0.72 0.02 (0.00, 0.04) –0.01 (–0.01, 0.00) 0.02 (0.00, 0.04) 
Male 50–62 c-statistic 0.63 0.01 (0.00, 0.03) 0.01 (0.01, 0.02)* 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.80 0.05 (0.02, 0.08)* 0.01 (–0.01, 0.02) 0.04 (0.01, 0.08)* 
Female 50–62 c-statistic 0.68 0.04 (0.02, 0.05)* 0.00 (0.00, 0.01) 0.03 (0.02, 0.05)* 
Male 28–49 R2 statistic 0.062 0.009 (–0.001, 0.020) –0.003 (–0.007, 0.000) 0.013 (0.003, 0.023)* 
Male 50–62 R2 statistic 0.027 0.007 (0.002, 0.011)* 0.003 (0.002, 0.005)* 0.003 (–0.001, 0.008) 
Female 28–49 R2 statistic 0.098 0.034 (0.015, 0.053)* 0.006 (–0.002, 0.014) 0.027 (0.001, 0.045)* 
Female 50–62 R2 statistic 0.044 0.017 (0.011, 0.023)* 0.001(–0.001, 0.004) 0.016 (0.010, 0.022)* 

* p < 0.05 (differences statistically significant with a two-tailed test).

† Multivariable risk score (MRS): MRS1 is based on regression coefficients from baseline-only Cox regression model and baseline risk factor values; MRS2 is based on regression coefficients from baseline-only Cox regression model and updated risk factor values; MRS3 is based on regression coefficients from time-dependent Cox regression model and updated risk factor values.

‡ Deviance, twice negative log-likelihood, by which lower values indicate better predictive ability; c-statistic, higher values indicate better predictive ability; R2 statistic,maximum-rescaled R2, by which higher values indicate better predictive ability.

§ This column shows the value of the given statistic (measure of predictive ability) for MRS3.

¶ Each of columns 5–7 compares two MRS models, with respect to the statistics in the third column, by showing the mean difference between the two models, across 1,000 bootstrap resamples, and the corresponding 95% bootstrap-based confidence interval.

TABLE 5.

Comparison of cross-validated predictive ability between coronary heart disease risk prediction models based on three types of multivariable risk scores,† by age and sex strata with 6-year follow-up, Framingham Heart Study, 1948–1978

Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 719 –9 (–22, 5) 0 (–7, 5) –8 (–21, 5) 
Male 50–62 Deviance 2,754 –41 (–59, –22)* 0 (–3, 3) –41 (–60, –23)* 
Female 28–49 Deviance 313 0 (–9, 8) 1 (–4, 6) –1 (–8, 6) 
Female 50–62 Deviance 2,493 –46 (–67, –22)* 0 (–4, 3) –46 (–67, –22)* 
Male 28–49 c-statistic 0.66 –0.02 (–0.04, 0.01) –0.01 (–0.03, 0.01) –0.01 (–0.03, 0.01) 
Male 50–62 c-statistic 0.64 0.00 (–0.01, 0.01) 0.00 (–0.01, 0.00) 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.75 0.01 (–0.03, 0.06) 0.00 (–0.03, 0.02) 0.02 (–0.02, 0.05) 
Female 50–62 c-statistic 0.69 0.00 (–0.01, 0.01) 0.00 (0.00, 0.00) 0.00 (–0.01, 0.01) 
Male 28–49 R2 statistic 0.029 –0.009 (–0.017, –0.001)* –0.003 (–0.008, 0.001) –0.006 (–0.014, 0.002) 
Male 50–62 R2 statistic 0.033 –0.002 (–0.007, 0.002) –0.002 (–0.003, 0.000) –0.001 (–0.005, 0.004) 
Female 28–49 R2 statistic 0.064 0.006 (–0.004, 0.018) –0.001 (–0.006, 0.004) 0.007 (–0.003, 0.018) 
Female 50–62 R2 statistic 0.047 0.000 (–0.005, 0.005) 0.000 (–0.001, 0.002) –0.001 (–0.005,0.004) 
Sex Age group (years) Measure of predictive ability‡ MRS3§ MRS3 – MRS1¶ MRS3 – MRS2¶ MRS2 – MRS1¶ 
Male 28–49 Deviance 719 –9 (–22, 5) 0 (–7, 5) –8 (–21, 5) 
Male 50–62 Deviance 2,754 –41 (–59, –22)* 0 (–3, 3) –41 (–60, –23)* 
Female 28–49 Deviance 313 0 (–9, 8) 1 (–4, 6) –1 (–8, 6) 
Female 50–62 Deviance 2,493 –46 (–67, –22)* 0 (–4, 3) –46 (–67, –22)* 
Male 28–49 c-statistic 0.66 –0.02 (–0.04, 0.01) –0.01 (–0.03, 0.01) –0.01 (–0.03, 0.01) 
Male 50–62 c-statistic 0.64 0.00 (–0.01, 0.01) 0.00 (–0.01, 0.00) 0.00 (–0.01, 0.01) 
Female 28–49 c-statistic 0.75 0.01 (–0.03, 0.06) 0.00 (–0.03, 0.02) 0.02 (–0.02, 0.05) 
Female 50–62 c-statistic 0.69 0.00 (–0.01, 0.01) 0.00 (0.00, 0.00) 0.00 (–0.01, 0.01) 
Male 28–49 R2 statistic 0.029 –0.009 (–0.017, –0.001)* –0.003 (–0.008, 0.001) –0.006 (–0.014, 0.002) 
Male 50–62 R2 statistic 0.033 –0.002 (–0.007, 0.002) –0.002 (–0.003, 0.000) –0.001 (–0.005, 0.004) 
Female 28–49 R2 statistic 0.064 0.006 (–0.004, 0.018) –0.001 (–0.006, 0.004) 0.007 (–0.003, 0.018) 
Female 50–62 R2 statistic 0.047 0.000 (–0.005, 0.005) 0.000 (–0.001, 0.002) –0.001 (–0.005,0.004) 

* p < 0.05 (differences statistically significant with a two-tailed test).

† Multivariable risk score (MRS): MRS1 is based on regression coefficients from baseline-only Cox regression model and baseline risk factor values; MRS2 is based on regression coefficients from baseline-only Cox regression model and updated risk factor values; MRS3 is based on regression coefficients from time-dependent Cox regression model and updated risk factor values.

‡ Deviance, twice negative log-likelihood, by which lower values indicate better predictive ability; c-statistics, higher values indicate better predictive ability; R2 statistic,maximum-rescaled R2, by which higher values indicate better predictive ability.

§ This column shows the value of the given statistic (measure of predictive ability) for MRS3.

¶ Each of columns 5–7 compares two MRS models, with respect to the statistics in the third column, by showing the mean difference between the two models, across 1,000 bootstrap resamples, and the corresponding 95% bootstrap-based confidence interval.

TABLE 6.

Comparison of cross-validated predictive ability between coronary heart disease risk prediction models based on four types of retrospectively based multivariable risk scores, by age and sex strata, Framingham Heart Study, 1948–1978

Sex Age group (years) Measure of predictive ability† Time lag between risk factor measurement and coronary heart disease event 
   Current‡ Current – 6 years§ Current – 10 years§ Current – 20 years§ 
Male 49–69 Deviance 710 –2 (–13, 9) –10 (–22, 1) –9 (–21, 3) 
Male 70–92 Deviance 338 –4 (–13, 6) –4 (–15, 6) –4 (–16, 10) 
Female 49–69 Deviance 644 4 (–6, 13) 1 (–10, 12) –7 (–18, 5) 
Female 70–92 Deviance 445 5 (–6, 16) 1 (–9, 11) 2 (–9, 14) 
Male 49–69 c-statistic 0.63 0.00 (–0.03, 0.03) 0.05 (0.02, 0.09)* 0.06 (0.02, 0.09)* 
Male 70–92 c-statistic 0.57 0.03 (–0.01, 0.09) 0.05 (–0.01, 0.11) 0.05 (–0.01, 0.11) 
Female 49–69 c-statistic 0.63 –0.01 (–0.04, 0.02) 0.00 (–0.03, 0.04) 0.06 (0.02, 0.10)* 
Female 70–92 c-statistic 0.53 –0.05 (–0.10, –0.01) * 0.00 (–0.04, 0.04) –0.01 (–0.06, 0.05) 
Male 49–69 R2 statistic 0.024 0.002 (–0.009, 0.015) 0.016 (0.006, 0.030)* 0.015 (0.004, 0.030)* 
Male 70–92 R2 statistic 0.009 0.007 (–0.001, 0.018) 0.008 (–0.001, 0.020) 0.008 (–0.004, 0.022) 
Female 49–69 R2 statistic 0.022 –0.005 (–0.015, 0.004) –0.003 (–0.015, 0.009) 0.012 (0.002, 0.024)* 
Female 70–92 R2 statistic 0.002 –0.009 (–0.018, –0.001)* 0.000 (–0.003, 0.002) 0.000 (–0.004, 0.004) 
Sex Age group (years) Measure of predictive ability† Time lag between risk factor measurement and coronary heart disease event 
   Current‡ Current – 6 years§ Current – 10 years§ Current – 20 years§ 
Male 49–69 Deviance 710 –2 (–13, 9) –10 (–22, 1) –9 (–21, 3) 
Male 70–92 Deviance 338 –4 (–13, 6) –4 (–15, 6) –4 (–16, 10) 
Female 49–69 Deviance 644 4 (–6, 13) 1 (–10, 12) –7 (–18, 5) 
Female 70–92 Deviance 445 5 (–6, 16) 1 (–9, 11) 2 (–9, 14) 
Male 49–69 c-statistic 0.63 0.00 (–0.03, 0.03) 0.05 (0.02, 0.09)* 0.06 (0.02, 0.09)* 
Male 70–92 c-statistic 0.57 0.03 (–0.01, 0.09) 0.05 (–0.01, 0.11) 0.05 (–0.01, 0.11) 
Female 49–69 c-statistic 0.63 –0.01 (–0.04, 0.02) 0.00 (–0.03, 0.04) 0.06 (0.02, 0.10)* 
Female 70–92 c-statistic 0.53 –0.05 (–0.10, –0.01) * 0.00 (–0.04, 0.04) –0.01 (–0.06, 0.05) 
Male 49–69 R2 statistic 0.024 0.002 (–0.009, 0.015) 0.016 (0.006, 0.030)* 0.015 (0.004, 0.030)* 
Male 70–92 R2 statistic 0.009 0.007 (–0.001, 0.018) 0.008 (–0.001, 0.020) 0.008 (–0.004, 0.022) 
Female 49–69 R2 statistic 0.022 –0.005 (–0.015, 0.004) –0.003 (–0.015, 0.009) 0.012 (0.002, 0.024)* 
Female 70–92 R2 statistic 0.002 –0.009 (–0.018, –0.001)* 0.000 (–0.003, 0.002) 0.000 (–0.004, 0.004) 

* p < 0.05 (differences statistically significant with a two-tailed test).

† Deviance, twice negative log-likelihood, by which lower values indicate better predictive ability; c-statistic, higher values indicate better predictive ability; R2 statistic,maximum-rescaled R2, by which higher values indicate better predictive ability.

‡ This column shows the value of the given statistic (measure of predictive ability) for the model based on current values.

§ Each of columns 5–7 compares two multivariable risk score models, with respect to the statistics in the third column, by showing the mean difference between the two models, across 1,000 bootstrap resamples, and the corresponding 95% bootstrap-based confidence interval.

References

1.
Anderson KM, Odell PM, Wilson PWF, et al. Cardiovascular disease risk profiles.
Am Heart J
 
1991
;
121
:
293
–8.
2.
Assmann G, Cullen P, Schulte H. Simple scoring scheme for calculating the risk of acute coronary events based on the 10-year follow-up of the Prospective Cardiovascular Munster (PROCAM) Study.
Circulation
 
2002
;
105
:
310
–15.
3.
Wilson PWF, D’Agostino RB, Levy D, et al. Prediction of coronary heart disease using risk factor categories.
Circulation
 
1998
;
97
:
1837
–47.
4.
Ramsay LE, Haq IU, Jackson PR, et al. Targeting lipid-lowering drug therapy for primary prevention of coronary disease: an updated Sheffield table.
Lancet
 
1996
;
348
:
387
–8.
5.
Tunstall-Pedoe H. The Dundee coronary risk-disk for management of change in risk factors.
BMJ
 
1991
;
303
:
744
–7.
6.
Shaper AG, Pocock SJ, Phillips AN, et al. Identifying men at high risk of heart attacks: strategy for use in general practice.
BMJ
 
(Clin Res Ed)
 
1986
;
293
:
474
–9.
7.
Prevention of coronary heart disease in clinical practice. Recommendations of the Second Joint Task Force of European and Other Societies on Coronary Prevention.
Eur Heart J
 
1998
;
19
:
1434
–503.
8.
Miettinen OS. Stratification by a multivariate confounder score.
Am J Epidemiol
 
1976
;
104
:
609
–20.
9.
Roman MJ, Shanker BA, Davis A, et al. Prevalence and correlates of accelerated atherosclerosis in systemic lupus erythematosus.
N Engl J Med
 
2003
;
349
:
2399
–406.
10.
Esdaile JM, Abrahamowicz M, Grodzicky T, et al. Traditional Framingham risk factors fail to fully account for accelerated atherosclerosis in systemic lupus erythematosus.
Arthritis Rheum
 
2001
;
44
:
2331
–7.
11.
Peduzzi P, Concato J, Kemper E, et al. A simulation study of the number of events per variable in logistic regression analysis.
J Clin Epidemiol
 
1996
;
49
:
1373
–9.
12. Prentice RL. Surrogate endpoints in clinical trials.
Stat Med
 
1989
;
8
:
431
–40.
13.
Psaty BM, Weiss NS, Furberg CD, et al. Surrogate end points, health outcomes, and the drug approval process for the treatment of risk factors for cardiovascular disease.
JAMA
 
1999
;
282
:
786
–90.
14.
Benfante R, Hwang LJ, Masaki K, et al. To what extent do cardiovascular risk factor values measured in elderly men represent their midlife values measured 25 years earlier? A preliminary report and commentary from the Honolulu Heart Program.
Am J Epidemiol
 
1994
;
140
:
206
–16.
15.
Zavaroni I, Ardigo D, Massironi P, et al. Do coronary heart disease risk factors change over time?
Metabolism
 
2002
;
51
:
1022
–6.
16.
Clarke R, Shipley M, Lewington S, et al. Underestimation of risk associations due to regression dilution in long-term follow-up of prospective studies.
Am J Epidemiol
 
1999
;
150
:
341
–53.
17.
D’Agostino RB, Russell MW, Huse DM, et al. Primary and subsequent coronary risk appraisal: new results from the Framingham Study.
Am Heart J
 
2000
;
139
:
272
–81.
18.
Wu M, Ware JH. On the use of repeated measurements in regression analysis with dichotomous responses.
Biometrics
 
1979
;
35
:
513
–21.
19.
Kawachi I, Colditz GA, Stampfer MJ, et al. Smoking cessation and time course of decreased risks of coronary heart disease in middle-aged women.
Arch Intern Med
 
1994
;
154
:
169
–75.
20.
Law MR, Wald NJ, Thompson SG. By how much and how quickly does reduction in serum cholesterol concentration lower risk of ischaemic heart disease?
BMJ
 
1994
;
308
:
367
–72.
21.
Downs JR, Clearfield M, Weiss S, et al. Primary prevention of acute coronary events with lovastatin in men and women with average cholesterol levels: results of AFCAPS/TexCAPS. Air Force/Texas Coronary Atherosclerosis Prevention Study.
JAMA
 
1998
;
279
:
1615
–22.
22.
Pekkanen J, Tervahauta M, Nissinen A, et al. Does the predictive value of baseline coronary risk factors change over a 30-year follow-up?
Cardiology
 
1993
;
82
:
181
–90.
23.
Vasan RS, Massaro JM, Wilson PWF, et al. Antecedent blood pressure and risk of cardiovascular disease. The Framingham Heart Study.
Circulation
 
2002
;
105
:
48
–53.
24.
Wilson PWF, Hoeg JM, D’Agostino RB, et al. Cumulative effects of high cholesterol levels, high blood pressure, and cigarette smoking on carotid stenosis.
N Engl J Med
 
1997
;
337
:
516
–22.
25.
Nieto FJ, Diez-Roux A, Szklo M, et al. Short- and long-term prediction of clinical and subclinical atherosclerosis by traditional risk factors.
J Clin Epidemiol
 
1999
;
52
:
559
–67.
26.
Hakim AA, Curb JD, Burchfiel CM, et al. Screening for coronary heart disease in elderly men based on current and past cholesterol levels.
J Clin Epidemiol
 
1999
;
52
:
1257
–65.
27.
Wannamethee SG, Shaper AG, Whincup PH, et al. Role of risk factors for major coronary heart disease events with increasing length of follow up.
Heart
 
1999
;
81
:
374
–9.
28.
Menotti A, Lanti M. Coronary risk factors predicting early and late coronary deaths.
Heart
 
2003
;
89
:
19
–24.
29.
Cupples LA, D’Agostino RB, Anderson K, et al. Comparison of baseline and repeated measure covariate techniques in the Framingham Heart Study.
Stat Med
 
1988
;
7
:
205
–18.
30.
D’Agostino RB, Lee ML, Belanger AJ, et al. Relation of pooled logistic regression to time-dependent Cox regression analysis: the Framingham Heart Study.
Stat Med
 
1990
;
9
:
1501
–15.
31.
Cox DR. Regression models and life tables (with discussion).
J R Stat Soc (B)
 
1972
;
34
:
187
–220.
32.
Zeger SL, Liang KY. Longitudinal data analysis for discrete and continuous outcomes.
Biometrics
 
1986
;
42
:
121
–30.
33.
Liang KY, Zeger SL. Longitudinal data analysis using generalized linear models.
Biometrika
 
1986
;
73
:
13
–22.
34.
Kannel W, Wolf P, Garrison R, eds. The Framingham Study: an epidemiological investigation of cardiovascular disease. Washington, DC: US Department of Commerce, 1988.
35.
Abbott RD, Curb JD, Rodriguez BL, et al. Age-related changes in risk factor effects on the incidence of coronary heart disease.
Ann Epidemiol
 
2002
;
12
:
173
–81.
36.
Barrett-Connor E. Sex differences in coronary heart disease. Why are women so superior? The 1995 Ancel Keys Lecture.
Circulation
 
1997
;
95
:
252
–64.
37.
Franklin SS, Larson MG, Khan SA, et al. Does the relation of blood pressure to coronary heart disease risk change with aging? The Framingham Heart Study.
Circulation
 
2001
;
103
:
1245
–9.
38.
Kannel WB, Wilson PWF. Risk factors that attenuate the female coronary disease advantage.
Arch Intern Med
 
1995
;
155
:
57
–61.
39.
Wilson PWF, Castelli WP, Kannel WB. Coronary risk prediction in adults (the Framingham Heart Study).
Am J Cardiol
 
1987
;
59
:
91G
–4G.
40.
Central Patient Registry and Coordinating Centre for the Lipid Research Clinics. Reference manual for the Lipid Research Clinics Prevalence Study. Vols 1 and 2. Chapel Hill, NC: University of North Carolina, 1974.
41.
Abrahamowicz M, du Berger R, Grover S. Flexible modeling of the effects of serum cholesterol on coronary heart disease mortality.
Am J Epidemiol
 
1997
;
145
:
714
–29.
42.
Jennrich RI, Schluchter MD. Unbalanced repeated-measures models with structured covariance matrices.
Biometrics
 
1986
;
42
:
805
–20.
43.
Zeger SL, Liang KY. Models for longitudinal data: a generalized estimating equations approach.
Biometrics
 
1988
;
44
:
1049
–60.
44.
Nagelkerke NJD. A note on a general definition of the coefficient of determination.
Biometrika
 
1991
;
78
:
691
–2.
45.
Harrell FE, Califf RM, Pryor DB, et al. Evaluating the yield of medical tests.
JAMA
 
1982
;
247
:
2543
–6.
46.
Hanley JA, McNeil BJ. The meaning and use of the area under a receiver operating characteristic (ROC) curve.
Radiology
 
1982
;
143
:
29
–36.
47.
Efron B, Tibshirani RJ. An introduction to the bootstrap. New York, NY: Chapman & Hall, 1993.
48.
Harrell FE, Lee KL, Mark DB. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors.
Stat Med
 
1996
;
15
:
361
–87.
49.
de Vries CL, Feskens EJM, Coulander CL, et al. Repeated measurements of serum cholesterol and blood pressure in relation to long-term incidence of myocardial infarction.
Cardiolog
 y
1993
;
82
:
89
–99.
50.
Tornberg SA, Jacobson KFS, Eklund GA. Stability and validity of a single serum cholesterol measurement in a prospective cohort study.
Int J Epidemiol
 
1988
;
17
:
797
–803.
51.
Thompson SG, Pocock SJ. The variability of serum cholesterol measurements: implications for screening and monitoring.
J Clin Epidemiol
 
1990
;
43
:
783
–9.
52.
Liao Y, McGee DL, Cooper RS, et al. How generalizable are coronary risk prediction models? Comparison of Framingham and two national cohorts.
Am Heart J
 
1999
;
137
:
837
–45.
53.
Emond MJ, Zareba W. Prognostic value of cholesterol in women of different ages.
J Womens Health
 
1997
;
6
:
295
–307.
54.
Niccoli G, Iacoviello L, Cianflone D, et al. Coronary risk factors: new perspectives.
Int J Epidemiol
 
2001
;
30(suppl 1)
:
S41
–7.