Abstract

Background

Blood-based biomarker tests can potentially change the landscape of colorectal cancer (CRC) screening. We characterize the conditions under which blood test screening would be as effective and cost-effective as annual fecal immunochemical testing or decennial colonoscopy.

Methods

We used the 3 Cancer Information and Surveillance Modeling Network–Colon models to compare scenarios of no screening, annual fecal immunochemical testing, decennial colonoscopy, and a blood test meeting Centers for Medicare & Medicaid (CMS) coverage criteria (74% CRC sensitivity and 90% specificity). We varied the sensitivity to detect CRC (74%-92%), advanced adenomas (10%-50%), screening interval (1-3 years), and test cost ($25-$500). Primary outcomes included quality-adjusted life-years (QALY) gained from screening and costs for a US average-risk cohort of individuals aged 45 years.

Results

Annual fecal immunochemical testing yielded 125-163 QALY gained per 1000 at a cost of $3811-$5384 per person, whereas colonoscopy yielded 132-177 QALY gained at a cost of $5375-$7031 per person. A blood test with 92% CRC sensitivity and 50% advanced adenoma sensitivity yielded 117-162 QALY gained if used every 3 years and 133-173 QALY gained if used every year but would not be cost-effective if priced above $125 per test. If used every 3 years, a $500 blood test only meeting CMS coverage criteria yielded 83-116 QALY gained at a cost of $8559-$9413 per person.

Conclusion

Blood tests that only meet CMS coverage requirements should not be recommended to patients who would otherwise undergo screening by colonoscopy or fecal immunochemical testing because of lower benefit. Blood tests need higher advanced adenoma sensitivity (above 40%) and lower costs (below $125) to be cost-effective.

Colorectal cancer (CRC) is the third most common cause of cancer-related mortality among men and women in the United States (1,2). In the year 2023, approximately 153 020 individuals will be diagnosed with CRC, and 52 550 will die from the disease. Screening has reduced the burden of CRC (3,4). The US Preventive Services Taskforce (USPSTF) recommends that average-risk individuals begin CRC screening at age 45 years and endorses a range of screening modalities, including decennial screening colonoscopy, annual fecal immunochemical chemical test, and multitarget stool DNA (mt-sDNA) tests every 3 years (5). Colonoscopy is a screening and diagnostic test, with high sensitivity for detection of cancer and precancerous lesions (6). In the United States, colonoscopy is the most commonly used CRC screening test (63.9%), followed by stool-based tests (7). Yet adherence to CRC screening recommendations is suboptimal; in 2020, 72% of individuals aged 50 years and older were up to date with recommended CRC screening (8). Ongoing adherence to fecal immunochemical testing screening is also low; less than 25% of people participating in fecal immunochemical testing screening were screened annually over a 3-year period (9).

Recently, blood-based biomarkers to detect CRC have emerged as a potential screening modality, and multiple trials are underway to establish the performance of novel blood-based biomarker tests for CRC screening (10). Blood tests are appealing because they are minimally invasive and easily performed in a doctor’s office with minimal risk (11). The Centers for Medicare & Medicaid Services (CMS) released a coverage decision for CRC screening of individuals at average risk for CRC, which would reimburse a triennial blood test with a demonstrated sensitivity of at least 74% for detection of CRC and specificity of at least 90% (12). However, the current CMS coverage decision does not stipulate requirements for adenoma detection, particularly the detection of advanced adenomas that put a person at higher risk of CRC. Moreover, modeling studies project blood tests that just achieve the minimum requirements in CMS’s coverage decision to yield substantially fewer life-years gained than either annual fecal immunochemical testing or decennial colonoscopy (13,14); hence, they could harm those who would otherwise receive annual fecal immunochemical testing or decennial colonoscopy screening.

We performed a threshold analysis of test performance characteristics, interval, and costs using the 3 Cancer Information and Surveillance Modeling Network (CISNET)–CRC microsimulation models to characterize the conditions under which a blood test for CRC screening would be noninferior to the most common US screening regimens, decennial colonoscopy, or annual fecal immunochemical testing, on the basis of effectiveness and cost-effectiveness. We identify characteristics that would render blood tests as effective and cost-effective as the alternatives and how far blood tests would be from the cost-effectiveness frontier under multiple scenarios using the net monetary benefit measure.

Methods

Microsimulation models

Analyses used the 3 CRC models part of the CISNET-Colon group: the Microsimulation Screening Analysis-Colon model (MISCAN-Colon) (15,16), the Simulation Model of Colorectal Cancer (SimCRC) (17), and the Colorectal Cancer Simulated Population model for Incidence and Natural history (CRC-SPIN) model (18). These models have been used to inform screening recommendations by the USPSTF (17,19), and analyses used the same assumptions as those made for the 2021 USPSTF analysis (17), including simulation of recent increases in CRC incidence before age 50 years in the US population. For these models, advanced adenomas are defined as adenomas that are 10 mm or larger. More details about the models’ assumptions are available on the CISNET models profiles website (20).

Screening strategies

Each modeling group simulated a cohort of 10 million individuals from 45 years to death. Four screening strategies were included in base-case scenarios: no screening (used as a comparator), colonoscopy every 10 years and fecal immunochemical testing every year (2 strategies considered cost-effective in prior studies) (21,22), and a blood test every 3 years (following the current CMS coverage decision). We simulated screening from ages 45 to 75 years for all tests. We assumed that all individuals who received an abnormal noninvasive test (fecal immunochemical testing or blood) underwent follow-up colonoscopy. Those with a negative follow-up colonoscopy after a false-positive triage test resumed their original screening regimen 10 years after their colonoscopy. Individuals simulated to have adenomas detected at colonoscopy were assumed to follow surveillance guidelines until age 85 years (23). Individuals were assumed to fully adhere to screening, follow-up colonoscopy, and surveillance.

Test characteristics

Fecal immunochemical testing characteristics were set at values used in a prior modeling study for the USPSTF (17), whereas colonoscopy sensitivities were updated following a meta-analysis (24) (Supplementary Table 1, available online). Baseline performance characteristics for a blood test were based on the current CMS coverage decision requirements, namely, 74% CRC sensitivity and 90% specificity (12). CMS did not specify advanced adenoma sensitivity requirements; we therefore assumed 10% sensitivity to advanced adenoma as baseline (ie, advanced adenomas are found only because of lack of specificity).

Economic assumptions

Supplementary Table 2 (available online) contains the disutility and cost assumptions used in this study. Base-case analyses assumed a cost of $500 per blood test. Following a comparable cost-effectiveness analysis (21), we assumed no differential disutility from blood tests relative to fecal immunochemical testing, implying no differential loss in quality of life caused by the 2 tests. Disutility from screening tests, treatments, and screening complications were discounted from the benefit of screening. Costs included screening and treatment and were inflated to 2021 US dollars. Diagnostic colonoscopy costs were incurred by those who tested positive using noninvasive tests and those with symptom-detected CRC (Supplementary Table 2, available online). All costs and benefits were discounted at 3% per year.

Outcomes

The primary measures of effectiveness were life-years gained and quality-adjusted life-years (QALY) gained relative to no screening over the remaining lifetime of the simulated cohort. We computed the costs of CRC care (including treatment and screening) and net costs relative to no screening. We report the incremental cost-effectiveness ratio (ICER) for nondominated strategies. A strategy is nondominated (ie, efficient) if no other strategy (or combination of strategies) provides more QALY gained at a lower cost (25). We also report the net monetary benefit for all screening strategies relative to no screening using a willingness to pay of $100 000 per QALY gained—a benchmark widely used in the health technology assessment cost-effectiveness (CEA) literature (26). Unlike the ICER, the net monetary benefit measure (net monetary benefit = willingness to pay * QALY gained—net costs) can be used to rank all strategies on a single scale (27), which is useful in this analysis because blood-based tests and other novel tests were projected to not be cost-effective in prior analyses (21,22). The net monetary benefit of a screening strategy can be interpreted as the dollar value of screening accrued to society and is useful for comparing the value provided by novel CRC screening tests relative to the standard of care of CRC screening (ie, decennial colonoscopy and annual fecal immunochemical testing).

Threshold analyses

We developed a full factorial experimental design consisting of 900 scenarios to explore the conditions under which blood tests are projected to be noninferior to fecal immunochemical testing or colonoscopy on the basis of effectiveness and cost-effectiveness. Effectiveness noninferiority requires that the test provides at least the same benefit (QALY gained) as the comparator (ie, colonoscopy). Cost-effectiveness noninferiority is the same as nondominance. In this analysis, we varied 4 test characteristics: blood test sensitivity for CRC (74%, 83%, and 92%), sensitivity for advanced adenomas (10%, 20%, 30%, 40%, and 50%), screening interval (1, 2, and 3 years), and test cost ($25-$500 in increments of $25). These values were selected to cover the plausible characteristics of blood tests that will be developed in the near future. The range of CRC sensitivity spans the current CMS minimum threshold (74%) up to the sensitivity of the currently available mt-sDNA tests (92%) (28) and encompasses the CRC sensitivity of existing blood-based biomarker tests, such as the Shield test (83%) (29). Sensitivity to detect advanced adenomas spans the false-positive rate under the CMS coverage decision (10%), and the maximum sensitivity to advanced adenomas explored (50%) goes beyond the advanced adenoma sensitivity of any existing noninvasive test: sensitivity to advanced adenomas for mt-sDNA was 42% for the first-generation test (28), 43.4% for the next-generation mt-sDNA test (30), and 46% for a multitarget stool RNA test (31). We also examined shorter screening intervals than the current CMS coverage decision. Finally, we reduced the price of CRC blood tests from the assumed baseline price of approximately $500 (32), down to $25—the approximate price of fecal immunochemical testing tests (Supplementary Table 2, available online).

The specificity of blood tests was held at 90% across all analyses, under the expectation that blood test developers will set detection thresholds as to maximize sensitivity subject to CMS’s minimum specificity constraint. Importantly, the CMS coverage decision specified a minimum CRC specificity of 90% but did not mention precursor lesions, making it unclear whether adenoma detection in the absence of CRC is to be considered a false-positive. Consistent with assumptions of prior modeling studies (17,18), we assume that a false-positive blood test only occurs when neither CRC nor adenomas are detected at colonoscopy.

We evaluated the cost-effectiveness of blood test screening regimens for each of the 900 scenarios (unique combinations of blood test accuracy, costs, and test interval) for each of the 3 models. We identified the scenarios in which blood tests were not dominated by either colonoscopy or fecal immunochemical testing. We report the scenarios that would render blood testing to be considered cost-effective by at least 1 of the 3 models (with an ICER under $150 000 per QALY gained—the upper range of willingness to pay values commonly used in cost-effectiveness analyses) (26).

Scenario discovery

Because of the large number of scenarios considered, a strategy was needed to 1) identify the test characteristics that have the strongest effect on the cost-effectiveness of blood tests and 2) present a low-dimensional summary of our results to describe how far a blood test scenario is from the cost-effectiveness frontier as a function of test characteristics. We drew from the robust decision making (33,34) approach to accomplish both tasks. In particular, we use the third step in the method, called scenario discovery (35), to identify conditions under which blood test screening regimens would fail or succeed in matching or exceeding the net monetary benefit of colonoscopy (the most used screening method in the United States) without requiring a probabilistic characterization of the test characteristics varied.

To identify test characteristics with the biggest effect on the cost-effectiveness of a blood test, we trained a random forest on the predicted cost-effectiveness status (ie, cost-effective vs not cost-effective relative to colonoscopy and fecal immunochemical testing) for each scenario using 5 independent variables: the 4 test characteristics (CRC and advanced adenoma sensitivity, screening interval, and cost) and the natural history model. The goal of this step is to ensure that we select and present the most relevant test characteristics in our analysis. We then used the random forest’s ranking of variable importance, using the mean decrease in Gini coefficient (36,37), to identify the test characteristics most strongly associated with cost-effectiveness.

Next, we chose the 3 variables with the largest effect on the cost-effectiveness of a blood test and included these as independent variables in a locally weighted regression model (38) to generate a smooth approximation of the net monetary benefit of screening given the 3 independent variables in an evenly spaced 3-dimensional grid of 10 000 points. These projections are conditioned on a CRC sensitivity of 92% (the upper range of the threshold analysis) and are averaged across the 3 models. We created those projections for the upper range of CRC sensitivity because it was the least important variable in our experimental design to determine the cost-effectiveness of blood tests and because this is a best-case scenario for finding the blood tests to be cost-effective; hence, it helps reveal the maximum cost and minimum advanced adenoma sensitivity at which blood tests provide equivalent net monetary benefit relative to colonoscopy if blood tests achieve 92% CRC sensitivity. Finally, we computed contour lines on the basis of the smoothed net monetary benefit estimates. This approach produces a “map” that test developers and policy makers can use to assess the net monetary benefit provided by new blood tests or other noninvasive tests with test performance in the range explored in this study in the context of the net monetary benefit of colonoscopy, the leading US alternative.

Results

Base case

In the absence of screening, the models projected 32-36 CRC deaths over the remaining lifetime of 1000 individuals aged 45 years (Table 1) and an average cost of $5268-$5845 per person for CRC care. Under a program of annual fecal immunochemical testing screening, models projected 6-12 CRC deaths per 1000 individuals aged 45 years, yielding 125-163 QALY gained per 1000 individuals relative to no screening. Annual fecal immunochemical testing resulted in a cost of $3811-$5384 per person, including both CRC screening and treatment costs, making it a cost-saving strategy relative to no screening. Colonoscopy screening further reduced CRC deaths to 4-10, yielding 132-177 QALY gained per 1000 individuals relative to no screening at a cost of $5375-$7031 per person. The ICER of decennial colonoscopy relative to annual fecal immunochemical testing was $100 125-$250 971 per QALY gained. The net monetary benefit of both strategies was similar, valued at $12 763-$18 128 per person for fecal immunochemical testing and $11 773-$18 126 for colonoscopy at a $100 000 willingness to pay per QALY gained.

Table 1.

Cost-effectiveness results for comparator tests and select blood tests

Scenario classTest, intervalTest costaSensitivityb
CRC deathscQALY gainedc,dCosts ($)c,dNet monetary benefit ($) eICER ($)f
Advanced adenomaCRC
Base caseNo screeninghhh32-3605268-58450Dominatedg
Fecal immunochemical testing, 1 y23.420.2380.7386-12125-1633811-538412 763-18 128Reference strategyM, C, S
Colonoscopy, 10 y963.95-1312.36i0.910.914-10132-1775375-703111 773-18 126100 125-250 971M, C, S
Blood, 3 y5000.10.7414-1883-1168559-94134853-8657Dominatedg
No change in blood test costBlood, 3 y5000.10.9213-1786-1258636-94105251-9578Dominatedg
Blood, 3 y5000.50.747-13113-1586850-84558516-14 619Dominatedg
Blood, 3 y5000.50.926-13117-1626868-84568876-15 044Dominatedg-
Blood, 1 y5000.10.747-12119-15411 282-12 6344873-9593Dominatedg
Blood, 1 y5000.50.924-10133-17310 024-11 5607388-13 022Dominatedg
3 models project blood test as cost-effectivejBlood, 1 y500.40.835-10131-1714602-620712 515-18 24453 740-143 193M, C, S
Blood, 1 y500.40.925-10131-1724602-621112 586-18 30161 830-127 195M, C, S
Blood, 1 y500.50.744-10132-1724548-613912 660-18 42561 472-115 743M, C, S
Blood, 1 y500.50.834-10132-1734547-613312 743-18 48463 510-95 272M, C, S
Blood, 1 y500.50.924-10133-1734549-613212 816-18 53467 399-112 977M, C, S
Scenario classTest, intervalTest costaSensitivityb
CRC deathscQALY gainedc,dCosts ($)c,dNet monetary benefit ($) eICER ($)f
Advanced adenomaCRC
Base caseNo screeninghhh32-3605268-58450Dominatedg
Fecal immunochemical testing, 1 y23.420.2380.7386-12125-1633811-538412 763-18 128Reference strategyM, C, S
Colonoscopy, 10 y963.95-1312.36i0.910.914-10132-1775375-703111 773-18 126100 125-250 971M, C, S
Blood, 3 y5000.10.7414-1883-1168559-94134853-8657Dominatedg
No change in blood test costBlood, 3 y5000.10.9213-1786-1258636-94105251-9578Dominatedg
Blood, 3 y5000.50.747-13113-1586850-84558516-14 619Dominatedg
Blood, 3 y5000.50.926-13117-1626868-84568876-15 044Dominatedg-
Blood, 1 y5000.10.747-12119-15411 282-12 6344873-9593Dominatedg
Blood, 1 y5000.50.924-10133-17310 024-11 5607388-13 022Dominatedg
3 models project blood test as cost-effectivejBlood, 1 y500.40.835-10131-1714602-620712 515-18 24453 740-143 193M, C, S
Blood, 1 y500.40.925-10131-1724602-621112 586-18 30161 830-127 195M, C, S
Blood, 1 y500.50.744-10132-1724548-613912 660-18 42561 472-115 743M, C, S
Blood, 1 y500.50.834-10132-1734547-613312 743-18 48463 510-95 272M, C, S
Blood, 1 y500.50.924-10133-1734549-613212 816-18 53467 399-112 977M, C, S
a

Centers for Medicare & Medicaid Services (CMS) cost per covered screening test (applied to those aged older than 65 years). Costs for blood tests, which are not currently covered by Centers for Medicare & Medicaid Services, were assumed to be similar to the cost of Epi proColon.32 See Supplementary Table 2 (available online) for detailed costs. CRC = colorectal cancer; CRCSPIN = Colorectal Cancer Simulated Population model for Incidence and Natural history; MISCAN = Microsimulation Screening Analysis model; QALY = quality-adjusted life-years; SIMCRC = Simulation Model of Colorectal Cancer.

b

Sensitivity of colonoscopy refers to lesion-level sensitivity, whereas sensitivity for noninvasive tests refers to person-level sensitivity, which considers the most advanced lesion of a screened individual.

c

Outcomes presented as events or life-years per 1000 individuals aged 45 years.

d

Discounted at 3% per year.

e

Net monetary benefit per person setting a willingness to pay of $100 000 per QALY gained (ie, net monetary benefit = 100 000 * QALY gained—net costs). Net costs are the cost of the screening regimen minus the costs of the no screening regimen. In this context, the net monetary benefit can be interpreted as the average absolute monetary value for those who adhere to a screening regimen, accounting for all benefits and costs incorporated in the analysis. The difference between the net monetary benefit of the 2 strategies reveals the loss of benefit caused by choosing a suboptimal strategy.

f

Because no screening is a dominated strategy, incremental cost-effectiveness ratio (ICER) is not calculated for the least effective nondominated strategy. M, C, and S refer to MISCAN, CRCSPIN, and SIMCRC and are displayed if each of those models projects the test to be cost-effective (ie, strategy was not dominated and had an ICER lower than $150 000). ICER ranges reflect the range of ICERs for the models that projected a strategy not to be extended dominated.

g

ICER is not computed by any model because the screening strategy is dominated.

h

The no screening strategy has no test cost or sensitivity values.

i

Screening colonoscopy costs depend on whether lesions are removed during the exam. We provide the lower cost, without lesion removal, and the higher cost, with lesion removal. Further detail is provided in Supplementary Table 2 (available online).

j

This table presents only blood tests projected as cost-effective by the 3 models at a $50 cost. Supplementary Table 3 (available online) contains other combinations of test performance characteristics where 2 or 3 models project blood tests to be cost-effective.

Table 1.

Cost-effectiveness results for comparator tests and select blood tests

Scenario classTest, intervalTest costaSensitivityb
CRC deathscQALY gainedc,dCosts ($)c,dNet monetary benefit ($) eICER ($)f
Advanced adenomaCRC
Base caseNo screeninghhh32-3605268-58450Dominatedg
Fecal immunochemical testing, 1 y23.420.2380.7386-12125-1633811-538412 763-18 128Reference strategyM, C, S
Colonoscopy, 10 y963.95-1312.36i0.910.914-10132-1775375-703111 773-18 126100 125-250 971M, C, S
Blood, 3 y5000.10.7414-1883-1168559-94134853-8657Dominatedg
No change in blood test costBlood, 3 y5000.10.9213-1786-1258636-94105251-9578Dominatedg
Blood, 3 y5000.50.747-13113-1586850-84558516-14 619Dominatedg
Blood, 3 y5000.50.926-13117-1626868-84568876-15 044Dominatedg-
Blood, 1 y5000.10.747-12119-15411 282-12 6344873-9593Dominatedg
Blood, 1 y5000.50.924-10133-17310 024-11 5607388-13 022Dominatedg
3 models project blood test as cost-effectivejBlood, 1 y500.40.835-10131-1714602-620712 515-18 24453 740-143 193M, C, S
Blood, 1 y500.40.925-10131-1724602-621112 586-18 30161 830-127 195M, C, S
Blood, 1 y500.50.744-10132-1724548-613912 660-18 42561 472-115 743M, C, S
Blood, 1 y500.50.834-10132-1734547-613312 743-18 48463 510-95 272M, C, S
Blood, 1 y500.50.924-10133-1734549-613212 816-18 53467 399-112 977M, C, S
Scenario classTest, intervalTest costaSensitivityb
CRC deathscQALY gainedc,dCosts ($)c,dNet monetary benefit ($) eICER ($)f
Advanced adenomaCRC
Base caseNo screeninghhh32-3605268-58450Dominatedg
Fecal immunochemical testing, 1 y23.420.2380.7386-12125-1633811-538412 763-18 128Reference strategyM, C, S
Colonoscopy, 10 y963.95-1312.36i0.910.914-10132-1775375-703111 773-18 126100 125-250 971M, C, S
Blood, 3 y5000.10.7414-1883-1168559-94134853-8657Dominatedg
No change in blood test costBlood, 3 y5000.10.9213-1786-1258636-94105251-9578Dominatedg
Blood, 3 y5000.50.747-13113-1586850-84558516-14 619Dominatedg
Blood, 3 y5000.50.926-13117-1626868-84568876-15 044Dominatedg-
Blood, 1 y5000.10.747-12119-15411 282-12 6344873-9593Dominatedg
Blood, 1 y5000.50.924-10133-17310 024-11 5607388-13 022Dominatedg
3 models project blood test as cost-effectivejBlood, 1 y500.40.835-10131-1714602-620712 515-18 24453 740-143 193M, C, S
Blood, 1 y500.40.925-10131-1724602-621112 586-18 30161 830-127 195M, C, S
Blood, 1 y500.50.744-10132-1724548-613912 660-18 42561 472-115 743M, C, S
Blood, 1 y500.50.834-10132-1734547-613312 743-18 48463 510-95 272M, C, S
Blood, 1 y500.50.924-10133-1734549-613212 816-18 53467 399-112 977M, C, S
a

Centers for Medicare & Medicaid Services (CMS) cost per covered screening test (applied to those aged older than 65 years). Costs for blood tests, which are not currently covered by Centers for Medicare & Medicaid Services, were assumed to be similar to the cost of Epi proColon.32 See Supplementary Table 2 (available online) for detailed costs. CRC = colorectal cancer; CRCSPIN = Colorectal Cancer Simulated Population model for Incidence and Natural history; MISCAN = Microsimulation Screening Analysis model; QALY = quality-adjusted life-years; SIMCRC = Simulation Model of Colorectal Cancer.

b

Sensitivity of colonoscopy refers to lesion-level sensitivity, whereas sensitivity for noninvasive tests refers to person-level sensitivity, which considers the most advanced lesion of a screened individual.

c

Outcomes presented as events or life-years per 1000 individuals aged 45 years.

d

Discounted at 3% per year.

e

Net monetary benefit per person setting a willingness to pay of $100 000 per QALY gained (ie, net monetary benefit = 100 000 * QALY gained—net costs). Net costs are the cost of the screening regimen minus the costs of the no screening regimen. In this context, the net monetary benefit can be interpreted as the average absolute monetary value for those who adhere to a screening regimen, accounting for all benefits and costs incorporated in the analysis. The difference between the net monetary benefit of the 2 strategies reveals the loss of benefit caused by choosing a suboptimal strategy.

f

Because no screening is a dominated strategy, incremental cost-effectiveness ratio (ICER) is not calculated for the least effective nondominated strategy. M, C, and S refer to MISCAN, CRCSPIN, and SIMCRC and are displayed if each of those models projects the test to be cost-effective (ie, strategy was not dominated and had an ICER lower than $150 000). ICER ranges reflect the range of ICERs for the models that projected a strategy not to be extended dominated.

g

ICER is not computed by any model because the screening strategy is dominated.

h

The no screening strategy has no test cost or sensitivity values.

i

Screening colonoscopy costs depend on whether lesions are removed during the exam. We provide the lower cost, without lesion removal, and the higher cost, with lesion removal. Further detail is provided in Supplementary Table 2 (available online).

j

This table presents only blood tests projected as cost-effective by the 3 models at a $50 cost. Supplementary Table 3 (available online) contains other combinations of test performance characteristics where 2 or 3 models project blood tests to be cost-effective.

A blood test with the minimum performance criteria put forth by CMS would be more costly and less effective than either decennial colonoscopy or annual fecal immunochemical testing. If used every 3 years, a blood test meeting these minimal performance characteristics would result in 14-18 CRC deaths and 83-116 QALY gained per 1000 individuals at a total cost of $8559-$9413 per person. Although the net monetary benefit of the test is positive at $4853-$8657, making the test cost-effective relative to no screening, this triennial test regimen would yield 34%-50% fewer QALY gained relative to decennial colonoscopy, resulting in a loss of 52%-70% of the net monetary benefit provided by decennial colonoscopy.

Threshold analyses

Effectiveness thresholds

A blood test with a 1- or 2-year screening interval and relatively high advanced adenoma sensitivity could be noninferior to annual fecal immunochemical testing in terms of life-years gained (Figure 1 A) and QALY gained (Figure 1 B) for some combinations of interval and advanced adenoma sensitivity. For instance, an annual blood test regimen with 74% CRC sensitivity and 20% advanced adenoma sensitivity was noninferior to annual fecal immunochemical testing in terms of QALY gained, but a biennial blood test regimen with 74% CRC sensitivity would require nearly 50% advanced adenoma sensitivity to be noninferior to fecal immunochemical testing. A triennial blood test regimen with the best accuracy assumptions (92% CRC sensitivity and 50% advanced adenoma sensitivity) nearly achieved QALY gained noninferiority, yielding 117-162 QALY gained per 1000 individuals, but models still projected a 1%-7% effectiveness gap relative to fecal immunochemical testing and an 8%-14% QALY gained gap relative to colonoscopy. Increasing a blood test’s sensitivity to advanced adenomas from 10% to 20% increased effectiveness more than increasing CRC sensitivity from 74% to 92% across the 3 models, but the marginal benefit of advanced adenoma sensitivity improvements decreases at higher sensitivity levels. Of the 3 models, 2 (CRCSPIN and SimCRC) projected that blood test screening strategies provided inferior QALY gained relative to decennial colonoscopy in all scenarios.

Effectiveness of blood colorectal cancer (CRC) screening tests as a function of sensitivity and interval. Each curve represents projected lifetime benefits per 1000 guideline-adherent individuals aged 45 years for alternative blood-based screening regimens as a function of blood tests’ sensitivity to CRC and advanced adenomas. Horizontal lines represent the projected benefits of annual fecal immunochemical testing and decennial colonoscopy. Noninvasive tests’ sensitivity is modeled as sensitivity to the most advanced lesion in the colon. Outcomes are discounted at 3% per year. All screening regimens start at 45 years and end at 75 years. Panel A presents discounted life-years gained from screening per 1000 individuals aged 45 years. Panel B presents quality-adjusted life-years gained from screening per 1000 individuals aged 45 years. Vertical axes vary across models to facilitate visualization of relative differences. AA = advance adenoma; COL = colonoscopy; FIT = fecal immunochemical testing; LYG = life-years gained; QALYG = quality-adjusted life-years gained.
Figure 1.

Effectiveness of blood colorectal cancer (CRC) screening tests as a function of sensitivity and interval. Each curve represents projected lifetime benefits per 1000 guideline-adherent individuals aged 45 years for alternative blood-based screening regimens as a function of blood tests’ sensitivity to CRC and advanced adenomas. Horizontal lines represent the projected benefits of annual fecal immunochemical testing and decennial colonoscopy. Noninvasive tests’ sensitivity is modeled as sensitivity to the most advanced lesion in the colon. Outcomes are discounted at 3% per year. All screening regimens start at 45 years and end at 75 years. Panel A presents discounted life-years gained from screening per 1000 individuals aged 45 years. Panel B presents quality-adjusted life-years gained from screening per 1000 individuals aged 45 years. Vertical axes vary across models to facilitate visualization of relative differences. AA = advance adenoma; COL = colonoscopy; FIT = fecal immunochemical testing; LYG = life-years gained; QALYG = quality-adjusted life-years gained.

Cost-effectiveness thresholds

None of the models projected that blood testing would be cost-effective without substantial reductions in price and higher sensitivity to advanced adenomas (Figure 2, Table 1). An annual blood test regimen with 50% advanced adenoma sensitivity and 92% CRC sensitivity was noninferior in effectiveness relative to annual fecal immunochemical testing. However, it would cost $10 024-$11 560 per person for a $500 blood test—well above the costs of either fecal immunochemical testing ($3811-$5384 per person) or decennial colonoscopy ($5375-$7031 per person) without yielding higher benefit.

Cost-effectiveness frontier for blood tests with varying test performance. The triangle, circle, and square shapes represent the projected performance of no screening, annual fecal immunochemical testing, and decennial colonoscopy in the cost-effectiveness plane. The line connecting fecal immunochemical testing and colonoscopy is the cost-effectiveness frontier (ie, the line connecting all strategies that provide good value for money). Diamond shapes connected by lines represent the performance of blood testing with varying levels of colorectal cancer (CRC) and advanced adenoma sensitivity, with solid lines representing the lower end of the simulated CRC sensitivity range (0.74) and dashed lines representing the higher end of the simulated CRC sensitivity range (0.92). Separate panels present projections from each of the 3 Cancer Information and Surveillance Modeling Network models. AA = advanced adenoma; COL = colonoscopy; FIT = fecal immunochemical testing; QALYG = quality-adjusted life-years gained.
Figure 2.

Cost-effectiveness frontier for blood tests with varying test performance. The triangle, circle, and square shapes represent the projected performance of no screening, annual fecal immunochemical testing, and decennial colonoscopy in the cost-effectiveness plane. The line connecting fecal immunochemical testing and colonoscopy is the cost-effectiveness frontier (ie, the line connecting all strategies that provide good value for money). Diamond shapes connected by lines represent the performance of blood testing with varying levels of colorectal cancer (CRC) and advanced adenoma sensitivity, with solid lines representing the lower end of the simulated CRC sensitivity range (0.74) and dashed lines representing the higher end of the simulated CRC sensitivity range (0.92). Separate panels present projections from each of the 3 Cancer Information and Surveillance Modeling Network models. AA = advanced adenoma; COL = colonoscopy; FIT = fecal immunochemical testing; QALYG = quality-adjusted life-years gained.

The conditions under which a blood test would be cost-effective varied across models (Table 1). For example, all models projected that a $50 blood test with 50% advanced adenoma sensitivity and 92% CRC would be cost-effective, yielding 133-173 QALY gained at a cost of $4549-$6132 per person. Such a test would yield an ICER of $67 399-$112 977 per QALY gained. Table 1 provides other combinations of performance characteristics of a $50 blood test for which all 3 models projected blood testing to be nondominated and cost-effective with a willingness to pay under $150 000. There were also scenarios where only 1 or 2 models projected blood testing to be cost-effective (Supplementary Table 3, available online). The maximum unit cost at which blood testing was cost-effective for at least 1 of the models (MISCAN) was $125. At this price, the test would need above 50% advanced adenoma sensitivity, above 83% CRC sensitivity, and to be used annually. Conversely, a test with 30% advanced adenoma sensitivity was only cost-effective per 2 models if priced at $25 per test. Supplementary Table 3 (available online) provides additional outcomes (life-years gained and CRC cases) along with the full set of 44 scenarios for which 1, 2, or the 3 models project blood tests to be cost-effective.

Net monetary benefit non-inferiority scenarios

The random forest trained to predict the cost-effectiveness of blood screening strategies relative to colonoscopy resulted in the following ranking of variable importance: 1) test cost, 2) advanced adenoma sensitivity, 3) test interval, 4) model, and 5) CRC sensitivity, confirming the small relative contribution of improved CRC sensitivity vs higher advanced adenoma sensitivity over the range we explored. Figure 3 shows the net monetary benefit for blood test screening regimens as a function of the 3 most important test characteristic inputs (test cost, advanced adenoma sensitivity, and test interval), averaged across the 3 models and assuming tests have 92% CRC sensitivity. A $500 test with 10% advanced adenoma sensitivity provided inferior net monetary benefit compared with colonoscopy, regardless of the screening frequency. Across the 3 interval scenarios, the test did not provide more than $8000 of net monetary benefit, whereas colonoscopy was estimated to provide $15 400 of net monetary benefit. Second, if a blood test were to achieve advanced adenoma sensitivity of 40%-50%, but cost $500 per test, blood testing would still fall approximately 20% short of matching the net monetary benefit of colonoscopy. Finally, if blood tests were offered at a cost lower than $125 per test and were to achieve advanced adenoma sensitivity of 40% or higher, the net monetary benefit of blood testing could be above $14 000, closing the net monetary benefit gap to approximately $1000 (or 7% of the net monetary benefit of colonoscopy).

Blood test net monetary benefit as a function of performance, interval, and costs. Colors estimate the average net monetary benefit of blood tests across models as a function of test (Centers for Medicare & Medicaid Services) cost, sensitivity to advanced adenomas, and test interval. Colorectal cancer (CRC) sensitivity is set at 92% (upper range of the experimental design). Each panel represents 1 blood-based screening interval. White dashed contour lines represent parameter combinations at which the net monetary benefit is constant, and the black dashed line represents the average net monetary benefit of decennial colonoscopy across the 3 models ($15 479 per person). This figure can be used to identify conditions under which blood tests provide equal or superior net monetary benefit relative to decennial colonoscopy (ie, the area under the black dashed line) and to estimate the gap between the net monetary benefit provided by blood-based tests and colonoscopy. For example, a blood-based test that costs $300 and has 10% advanced adenoma sensitivity yields a $10 000 net monetary benefit, which is approximately two-thirds of the net monetary benefit of colonoscopy. Further, a $500 blood test that is used every 3 years will still be approximately $3000 (ie, 20%) short of matching the net monetary benefit of colonoscopy even if it has 50% advanced adenoma sensitivity and 92% CRC sensitivity. For reference, the average net monetary benefit of annual fecal immunochemical testing estimated by the models is $15 815. COL = colonoscopy; NMB = net monetary benefit.
Figure 3.

Blood test net monetary benefit as a function of performance, interval, and costs. Colors estimate the average net monetary benefit of blood tests across models as a function of test (Centers for Medicare & Medicaid Services) cost, sensitivity to advanced adenomas, and test interval. Colorectal cancer (CRC) sensitivity is set at 92% (upper range of the experimental design). Each panel represents 1 blood-based screening interval. White dashed contour lines represent parameter combinations at which the net monetary benefit is constant, and the black dashed line represents the average net monetary benefit of decennial colonoscopy across the 3 models ($15 479 per person). This figure can be used to identify conditions under which blood tests provide equal or superior net monetary benefit relative to decennial colonoscopy (ie, the area under the black dashed line) and to estimate the gap between the net monetary benefit provided by blood-based tests and colonoscopy. For example, a blood-based test that costs $300 and has 10% advanced adenoma sensitivity yields a $10 000 net monetary benefit, which is approximately two-thirds of the net monetary benefit of colonoscopy. Further, a $500 blood test that is used every 3 years will still be approximately $3000 (ie, 20%) short of matching the net monetary benefit of colonoscopy even if it has 50% advanced adenoma sensitivity and 92% CRC sensitivity. For reference, the average net monetary benefit of annual fecal immunochemical testing estimated by the models is $15 815. COL = colonoscopy; NMB = net monetary benefit.

Discussion

Blood-based biomarkers for CRC screening must have substantially better test accuracy than the CMS coverage decision to achieve effectiveness similar to either annual fecal immunochemical testing or decennial colonoscopy and must cost considerably less to be cost-effective. Multiple combinations of test characteristics may render blood tests cost-effective, but all of them required the cost of the blood test to be $125 or lower and advanced adenoma sensitivity to be 30% or higher, and higher accuracy was needed to justify higher prices. For instance, 50% sensitivity to advanced adenoma was needed to justify a $125 blood test, whereas a blood test with 30% advanced adenoma sensitivity would have to be priced at $25 per test to be cost-effective. This result underscores the value of higher sensitivity to precursor lesions because their detection and removal prevents cancer and reduces health costs associated with cancer care.

Net monetary benefit estimates provide an objective measure of the value of screening to society and can be used to compare alternatives on a single scale. By this metric, if blood tests cost $500, only meet minimum CMS accuracy criteria, and are used every 3 years, their adoption could eliminate 52%-70% of the net monetary benefit afforded by colonoscopy for individuals who would otherwise follow colonoscopy screening. In contrast, annual fecal immunochemical testing’s net monetary benefit is on par with colonoscopy. Although there may be valid reasons to make blood tests available for CRC screening, they are unlikely to justify such a loss of benefit among individuals who would otherwise undergo colonoscopy screening. Beyond quantifying differences in long-term outcomes for different blood test screening regimens and currently recommended fecal immunochemical testing and colonoscopy regimens, this study identifies the characteristics of blood tests that would reduce or eliminate the potential loss caused by the adoption of blood tests. Even if blood tests cannot yet achieve the performance and cost thresholds we identified, this study can be used to quantify the gap between the value afforded by blood tests and colonoscopy.

Our findings align with previously published analyses but provide further information for policy makers and blood test developers. One study found that blood tests that only met the minimum CRC sensitivity from CMS and had low advanced adenoma sensitivity would frequently yield inferior results compared with fecal immunochemical testing and mt-sDNA (39). A second study explored multiple hypothetical tests that met or exceeded the CMS sensitivity and showed that a blood test with higher sensitivity for advanced adenoma would yield better benefits but did not consider cost-effectiveness (40). This analysis varied 4 test characteristics—test interval, sensitivity to CRC, sensitivity to advanced adenomas, and unit price—that developers can aim for with the goal of providing an effective and cost-effective new CRC screening modality. Still, we held several inputs constant in this analysis, including the specificity of the blood test, and we only included annual (vs biennial) fecal immunochemical testing as a comparator. Future research may include a wider range of comparators, including a fecal immunochemical test with a lower detection threshold calibrated to have 90% specificity.

Importantly, we assumed perfect adherence to primary screening and follow-up colonoscopy across the different screening regimens and intervals. A thorough evaluation of imperfect adherence would require considering differential adherence to fecal immunochemical testing and colonoscopy screening and real-world adherence to follow-up colonoscopy. Although data on follow-up colonoscopy adherence are scant, the existing data justify the concern that blood-based CRC screening might entail low adherence to follow-up colonoscopy. A randomized trial of patient adherence found higher adherence rates to screening for blood tests relative to fecal immunochemical testing but lower adherence rates to follow-up colonoscopy (41). Such an analysis would need to factor in trade-offs involved in improving adherence vs reducing the overall effectiveness of the screening program in the population and increasing costs. Finally, our analysis focused on blood tests that target CRC, as opposed to multicancer early detection tests. Future work may expand on our results to inform the adoption of novel blood-based biomarker tests for CRC screening.

Data availability

Data underlying figures and tables will be shared upon request.

Author contributions

Pedro Nascimento de Lima, PhD (Conceptualization; Formal analysis; Funding acquisition; Methodology; Software; Visualization; Writing—original draft; Writing—review & editing), Rosita van den Puttelaar, MS (Formal analysis; Methodology; Software; Writing—review & editing), Amy B. Knudsen, PhD (Formal analysis; Software; Writing—review & editing), Anne I. Hahn, MPH (Data curation; Project administration; Writing—original draft; Writing—review & editing), Karen M. Kuntz, PhD (Conceptualization; Supervision; Writing—review & editing), Jonathan Ozik, PhD (Funding acquisition; Software; Writing—review & editing), Nicholson Collier, PhD (Software; Writing—review & editing), Fernando Alarid-Escudero, PhD (Writing—review & editing), Ann G. Zauber, PhD (Funding acquisition; Writing—review & editing), John M. Inadomi, PhD (Conceptualization; Funding acquisition; Writing—review & editing), Iris Lansdorp-Vogelaar, PhD (Conceptualization; Writing—review & editing), and Carolyn M. Rutter, PhD (Conceptualization; Funding acquisition; Methodology; Supervision; Writing—original draft; Writing—review & editing).

Funding

This research was supported by grant U01-CA253913 from the National Cancer Institute (NCI) as part of the Cancer Intervention and Surveillance Modeling Network (CISNET). Additional funding was obtained through National Institutes of Health (NIH)/NCI Cancer Center Support grant P30 CA008748 (Zauber and Hahn). The funding agencies had no role in the study design, interpretation of results, or writing of the manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

Conflicts of interest

The authors declare no conflicts of interest.

Acknowledgements

This work was also supported in part by the US Department of Energy, Office of Science, under contract DE-AC0206CH11357. This research was completed with resources provided by the Laboratory Computing Resource Center at Argonne National Laboratory.

The funding agencies had no role in the study design, interpretation of results, or writing of the manuscript. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health.

References

1

Siegel
RL
,
Wagle
NS
,
Cercek
A
,
Smith
RA
,
Jemal
A.
Colorectal cancer statistics, 2023
.
CA Cancer J Clin
.
2023
;
73
(
3
):
233
-
254
. doi:

2

American Cancer Society
Key Statistics for Colorectal Cancer. https://www.cancer.org/cancer/types/colon-rectal-cancer/about/key-statistics.html. Accessed December 4,
2023
.

3

Edwards
BK
,
Ward
E
,
Kohler
BA
, et al.
Annual report to the nation on the status of cancer, 1975-2006, featuring colorectal cancer trends and impact of interventions (risk factors, screening, and treatment) to reduce future rates
.
Cancer
.
2010
;
116
(
3
):
544
-
573
. doi:

4

Zauber
AG
,
Winawer
SJ
,
O’Brien
MJ
, et al.
Colonoscopic polypectomy and long-term prevention of colorectal-cancer deaths
.
N Engl J Med
.
2012
;
366
(
8
):
687
-
696
. doi:

5

Davidson
KW
,
Barry
MJ
,
Mangione
CM
, et al. ;
US Preventive Services Task Force
.
Screening for colorectal cancer: US Preventive Services Task Force recommendation statement
.
JAMA
.
2021
;
325
(
19
):
1965
-
1977
., doi:

6

Issa
IA
,
Noureddine
M.
Colorectal cancer screening: an updated review of the available options
.
World J Gastroenterol
.
2017
;
23
(
28
):
5086
-
5096
. doi:

8

Bresalier
RS.
Colorectal cancer screening in a changing world
.
Gastroenterol Clin North Ame.
2022
;
51
(
3
):
577
-
591
. doi:

9

Fisher
DA
,
Princic
N
,
Miller-Wilson
LA
, et al.
Adherence to fecal immunochemical test screening among adults at average risk for colorectal cancer
.
Int J Colorectal Dis
.
2022
;
37
(
3
):
719
-
721
. doi:

10

Broccard
SP
,
Kasbi
AA
,
Bagaria
SP
,
Jones
J
,
Shoudry
M
,
Gabriel
EM.
Liquid biopsies for colorectal cancer: a narrative review of ongoing clinical trials and the current use of this technology at a comprehensive cancer center
.
J Gastrointest Oncol
.
2022
;
13
(
1
):
438
-
449
. doi:

11

Ganepola
GA
,
Nizin
J
,
Rutledge
JR
,
Chang
DH.
Use of blood-based biomarkers for early diagnosis and surveillance of colorectal cancer
.
World J Gastrointest Oncol
.
2014
;
6
(
4
):
83
-
97
. doi:

12

Centers for Medicare & Medicaid Services
. Screening for Colorectal Cancer—Blood-Based Biomarker Tests. https://www.cms.gov/medicare-coverage-database/view/ncacal-decision-memo.aspx?proposed=Y&NCAId=299. Accessed October 16,
2023
.

13

van den Puttelaar
R
,
Nascimento de Lima
P
,
Knudsen
AB
, et al.
CMS coverage decision: implications for cost-effectiveness of colorectal cancer screening
.
Gastroenterology
.
2023
. doi:

14

Aziz
Z
,
Wagner
S
,
Agyekum
A
, et al.
Cost-effectiveness of liquid biopsy for colorectal cancer screening in patients who are unscreened
.
JAMA Netw Open
.
2023
;
6
(
11
):
e2343392
. doi:

15

Loeve
F
,
Boer
R
,
van Oortmarssen
GJ
,
van Ballegooijen
M
,
Habbema
JD.
The MISCAN-COLON simulation model for the evaluation of colorectal cancer screening
.
Comput Biomed Res
.
1999
;
32
(
1
):
13
-
33
. doi:

16

van Hees
F
,
Habbema
JD
,
Meester
RG
,
Lansdorp-Vogelaar
I
,
van Ballegooijen
M
,
Zauber
AG.
Should colorectal cancer screening be considered in elderly persons without previous screening? A cost-effectiveness analysis
.
Ann Intern Med
.
2014
;
160
(
11
):
750
-
759
. doi:

17

Knudsen
AB
,
Rutter
CM
,
Peterse
EFP
, et al.
Colorectal cancer screening: an updated modeling study for the US Preventive Services Task Force
.
JAMA
.
2021
;
325
(
19
):
1998
-
2011
. doi:

18

Nascimento de Lima
P
,
Rutter
CM
,
Maerzluft
C
,
Ozik
J
,
Collier
N.
Robustness analysis of colorectal cancer colonoscopy screening strategies
.
medRxiv
.
2023
. doi:

19

Knudsen
AB
,
Zauber
AG
,
Rutter
CM
, et al.
Estimation of benefits, burden, and harms of colorectal cancer screening strategies: modeling study for the US Preventive Services Task Force
.
JAMA
.
2016
;
325
(
19
):
1998
-
2011
. doi:

20

CISNET
. Colorectal Cancer Models Overview.
2024
. https://resources.cisnet.cancer.gov/registry/site-summary/colorectal/. Accessed February 1, 2024.

21

Peterse
EFP
,
Meester
RGS
,
de Jonge
L
, et al.
Comparing the Cost-Effectiveness of Innovative Colorectal Cancer Screening Tests
.
J Natl Cancer Inst
.
2021
;
113
(
2
):
154
-
161
. doi:

22

Ladabaum
U
,
Mannalithara
A.
Comparative effectiveness and cost effectiveness of a multitarget stool DNA test to screen for colorectal neoplasia
.
Gastroenterology
.
2016
;
151
(
3
):
427
-
439 e6
. doi:

23

Gupta
S
,
Lieberman
D
,
Anderson
JC
, et al.
Recommendations for follow-up after colonoscopy and polypectomy: a consensus update by the US Multi-Society Task Force on colorectal cancer
.
Gastroenterology
.
2020
;
158
(
4
):
1131
-
1153.e5
. doi:

24

Zhao
S
,
Wang
S
,
Pan
P
, et al.
Magnitude, risk factors, and factors associated with adenoma miss rate of tandem colonoscopy: a systematic review and meta-analysis
.
Gastroenterology
.
2019
;
156
(
6
):
1661
-
1674.e11
. doi:

25

Mark
DH.
Visualizing cost-effectiveness analysis
.
JAMA
.
2002
;
287
(
18
):
2428
-
2429
. doi:

26

Neumann
PJ
,
Kim
DD.
Cost-effectiveness thresholds used by study authors, 1990-2021
.
JAMA
.
2023
;
329
(
15
):
1312
-
1314
. doi:

27

Paulden
M.
Calculating and interpreting ICERs and net benefit
.
Pharmacoeconomics
.
2020
;
38
(
8
):
785
-
807
. doi:

28

Imperiale
TF
,
Ransohoff
DF
,
Itzkowitz
SH
, et al.
Multitarget stool DNA testing for colorectal-cancer screening
.
N Engl J Med
.
2014
;
370
(
14
):
1287
-
1297
. doi:

29

Chung
DC
,
Gray
DM
2nd
,
Singh
H
, et al.
A cell-free DNA blood-based test for colorectal cancer screening
.
N Engl J Med
.
2024
;
390
(
11
):
973
-
983
. doi:

30

Imperiale
TF
,
Porter
K
,
Zella
J
, et al. ;
BLUE-C Study Investigators
.
Next-generation multitarget stool DNA test for colorectal cancer screening
.
N Engl J Med
.
2024
;
390
(
11
):
984
-
993
. doi:

31

Barnell
EK
,
Wurtzler
EM
,
La Rocca
J
, et al.
Multitarget stool RNA test for colorectal cancer screening
.
JAMA
.
2023
;
330
(
18
):
1760
-
1768
. doi:

33

Lempert
RJ.
Robust Decision Making (RDM)
. In: Marchau VAWJ, Walker WE, Bloemen PJTM, Popper SW, eds.
Decision Making under Deep Uncertainty.
Cham, Switzerland: Springer;
2019
;
23
-
51
.

34

Nascimento de Lima
P.
Robust Decision Making in Health Policy: Applications to COVID-19 and Colorectal Cancer
.
Pardee RAND Graduate School
;
2022
.

35

Bryant
BP
,
Lempert
RJ.
Thinking inside the box: a participatory, computer-assisted approach to scenario discovery
.
Technol Forecast Soc Change
.
2010
;
77
(
1
):
34
-
49
. doi:

36

Breiman
L.
Random forests
.
Mach Learn
.
2001
;
45
(
1
):
5
-
32
. doi:

37

Liaw
A
,
Wiener
M
.
Classification and regression by randomForest
.
R News
.
2002
;
2
:
18
-
22
.

38

Cleveland
WS
,
Grosse
E
,
Shyu
WM
. Local regression models. In:
Chambers JM, Hastie TJ
, eds.
Statistical Models in S
.
Wadsworth & Brooks/Cole
;
1992
.

39

Fendrick
AM
,
Vahdat
V
,
Chen
JV
, et al.
Comparison of simulated outcomes between stool- and blood-based colorectal cancer screening tests
.
Popul Health Manag
.
2023
;
26
(
4
):
239
-
245
. doi:

40

Piscitello
A
,
Carroll
LN
,
Fransen
S
, et al.
Differential impact of test performance characteristics on burden-to-benefit tradeoffs for blood-based colorectal cancer screening: a microsimulation analysis
.
J Med Screen
.
2023
;
30
(
4
):
175
-
183
. doi:

41

Coronado
GD
,
Jenkins
CL
,
Shuster
E
, et al.
Blood-based colorectal cancer screening in an integrated health system: a randomised trial of patient adherence
.
Gut
.
2024
;
73
(
4
):
622
-
628
. doi:

This is an Open Access article distributed under the terms of the Creative Commons Attribution License (https://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.

Supplementary data