A standardised, generic, validated approach to stratify the magnitude of clinical benefit that can be anticipated from anti-cancer therapies: the European Society for Medical Oncology Magnitude of Clinical Benefit Scale (ESMO-MCBS)

Abstract

The value of any new therapeutic strategy or treatment is determined by the magnitude of its clinical benefit balanced against its cost. Evidence for clinical benefit from new treatment options is derived from clinical research, in particular phase III randomised trials, which generate unbiased data regarding the efficacy, benefit and safety of new therapeutic approaches. To date, there is no standard tool for grading the magnitude of clinical benefit of cancer therapies, which may range from trivial (median progression-free survival advantage of only a few weeks) to substantial (improved long-term survival). Indeed, in the absence of a standardised approach for grading the magnitude of clinical benefit, conclusions and recommendations derived from studies are often hotly disputed and very modest incremental advances have often been presented, discussed and promoted as major advances or ‘breakthroughs’. Recognising the importance of presenting clear and unbiased statements regarding the magnitude of the clinical benefit from new therapeutic approaches derived from high-quality clinical trials, the European Society for Medical Oncology (ESMO) has developed a validated and reproducible tool to assess the magnitude of clinical benefit for cancer medicines, the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS). This tool uses a rational, structured and consistent approach to derive a relative ranking of the magnitude of clinically meaningful benefit that can be expected from a new anti-cancer treatment. The ESMO-MCBS is an important first step to the critical public policy issue of value in cancer care, helping to frame the appropriate use of limited public and personal resources to deliver cost-effective and affordable cancer care. The ESMO-MCBS will be a dynamic tool and its criteria will be revised on a regular basis.

ESMO, clinical benefit, tool

introduction

The value of any new therapeutic strategy or treatment is determined by the magnitude of its clinical benefit balanced against its cost [1]. Value considerations have become increasingly important in an era of rapid expansion of new, expensive cancer medicines and other technologies such as advanced radiotherapy techniques or robotic surgery which provide small incremental benefits [2–5] within the context of cost-constrained health care systems [6]. This is especially true in Europe where the costs of care delivery [6] and cancer outcomes [7–9] vary substantially across Europe with the latter being influenced by the level of economic development [9, 10]. In some instances, discrepant outcomes between countries in Europe can be attributed to inordinate delays, sometimes of years, in making highly effective treatments available at an affordable cost to the patient [11, 12].

Whereas costs of procurement and out of pocket expenditures vary from country to country, the magnitude of clinical benefit, as derived from well-designed clinical trials, is a relative constant. Consequently, meaningful discussion of value and relative value are predicated on an understanding of the magnitude of clinical benefit [1]. Clinical benefit in this context refers to the added benefit compared with a control which, in most cases, is the best current standard care.

Evidence for clinical benefit from new treatment approaches is derived from comparative outcome studies, most commonly phase III randomised clinical trials, which generate ostensibly unbiased data regarding the efficacy, benefit and safety of new therapeutic approaches. The potential benefits of a new treatment can be summarised as either living longer and/or living better, evaluated in clinical studies through the treatment effect on overall survival (OS) and/or quality of life (QoL), and their surrogates (Table 1). In studies of interventions with curative intent in which mature survival data are not yet available disease-free survival (DFS), recurrence-free survival (RFS), event-free survival (EFS), distant disease-free survival and time to recurrence (TTR), are used as surrogate measures. The validity of this approach, though not uncontroversial [13], is relatively well supported by data derived from a wide range of solid tumour settings including in colon [14], gastric [15], lung [16] and breast [17] cancers. In studies evaluating therapies in non-curative settings, progression-free survival (PFS), and time to progression (TTP) provide information about biological activity and may indicate benefit for some patients [18, 19]; however, they are not reliable surrogates for improved survival [18, 20–23] or QoL [23, 24].

Table 1.

Potential benefits of a new treatment

Living longer

Improved OS

Improved surrogate of OS

DFS (when OS data are immature in adjuvant setting)

Improved PFS

Living better

Improved quality of life

Improved surrogate of quality of life

Improved PFS

Reduced toxicity

To date, there is no standard tool for grading the magnitude of clinical benefit of cancer therapies [25, 26], which may range from trivial (median PFS advantage of only a few weeks) to substantial (improved long-term survival). Indeed, in the absence of a standardised approach for grading, the magnitude of clinical benefit, conclusions and recommendations derived from studies are often hotly disputed [25] and very modest incremental advances have often been presented, discussed and promoted as major advances or ‘breakthroughs’ [5, 25–29]. Overestimating or overstating the benefits from new intervention can cause harm: it confounds public policy decision making [29], undermines the credibility of oncology research reporting [26, 29, 30], harms patients who choose to undertake treatments based on exaggerated expectations that may subject them to either risk of adverse effects, inconvenience or substantial personal costs [26, 28] and, in the public domain, they fuel sometimes inappropriate hype or disproportionate expectations about novel treatments [31, 32] and the need to allocate public or personal funds to provide them.

It is important for the Oncology Community to present clear and unbiased statements regarding the magnitude of clinical benefit from new therapeutic approaches supported by credible research. ESMO aims to emphasize those treatments which bring substantial improvements to the duration of survival and/or the QoL of cancer patients which need to be distinguished from those whose benefits are more modest, limited or even marginal. To this end, ESMO has undertaken the development of a validated and reproducible tool to assess the magnitude of clinical benefit of anti-cancer interventions, the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS). ESMO intends to apply this scale prospectively to each new anti-cancer drug/intervention that will be European Medicines Agency (EMA) approved. Drugs or treatment interventions that obtain the highest scores on the scale will be emphasized in the ESMO guidelines, with the hope that they will be rapidly endorsed by health authorities across the European Union.

background and methodology

An ESMO Task Force to guide the development of the grading scale was established in March 2013. The members of the Task Force co-chaired by Elisabeth de Vries and Martine Piccart are Richard Sullivan, Nathan Cherny, Urania Dafni, Martijn Kerst, Alberto Sobrero and Christoph Zielinski. A first-generation draft scale was developed and adapted through a ‘snowball’ method based upon previous work of Task Force members who had independently developed preliminary models of clinical benefit grading. The first-generation scale was sent for review by 276 members of the ESMO faculty and a team of 51 expert biostatisticians.

The second-generation draft was formulated based on the feedback from faculty and biostatisticians and the conceptual work of Alberto Sobrero regarding the integration of both hazard ratio (HR), prognosis and absolute differences in data interpretation [33, 34]. The second-generation draft was applied in a wide range of contemporary and historical disease settings by members of the ESMO-MCBS Task Force, the ESMO Guidelines Committee and a range of invited experts. Results were scrutinised for face validity, coherence and consistency. Where deficiencies were observed or reported, targeted modifications were implemented and the process of field testing and review was repeated. This process was repeated through 13 redrafts of the scale preceding the current one (ESMO-MCBS v1.0). The final version and fielded testing results were reviewed by selected members of the ESMO faculty and the ESMO Executive Board.

The goal of the ESMO-MCBS evaluation was to assign the highest grade to trials having adequate power for a relevant magnitude of benefit, and to make appropriate grade adjustment to reflect the observed magnitude of benefit. To achieve this goal, a dual rule was implemented; first, taking into account the variability of the estimated HR from a study, the lower limit of the 95% confidence interval (CI) for the HR is compared with specified threshold values; and secondly the observed absolute difference in treatment outcomes is compared with the minimum absolute gain considered as beneficial. Different candidate threshold values for HR and absolute gains for survival, DFS and PFS, adjusted to represent as accurately as possible the expert opinion of the oncology community, have been explored through extensive simulations. The finally implemented combined thresholds for the HR and the minimum observed benefit that could be considered as deserving the highest grade in both the curative and non-curative setting are outlined in Table 2.

Table 2.

Maximal preliminary scores

Treatments with curative intent (form 1)

>5% improvement of survival at ≥3-year follow-up

Improvements in DFS alone HR <0.60 (primary end point) in studies without mature survival data

Treatments with non-curative intent (form 2)

Primary outcome OS (form 2a)

Control ≤12 months

HR ≤0.65 AND gain ≥3 months OR

Increase in 2-year survival alone ≥10%

Control >12 months

HR ≤0.70 AND gain ≥5 months OR

Increase in 3-year survival alone ≥10%

Primary outcome PFS (form 2b)

Control ≤6 months

HR ≤0.65 AND gain ≥1.5 months

Control >6 months

HR ≤0.65 AND gain ≥3 months

In all forms, HR thresholds refer to the lower extreme of the 95% CI (Figure 1). The performance of the evaluation rule based on the lower limit of the 95% CI of HR, was compared with the simpler rule of using a cut-off for the point estimate of HR, in conjunction with the additional rule on the minimum absolute gain in treatment outcome. The simulation results under different HR values and corresponding power, favoured the proposed approach to use the lower limit of the 95% CI which takes into account the variability of the estimate. The correspondence between an HR value and the minimum absolute gain considered as beneficial according to the ESMO-MCBS, is presented by median survival (OS or PFS) for standard treatment, in Figure 2. For example, for a standard treatment median survival of 6 months, an absolute gain of 3 months corresponds to an HR = 0.67, while a gain of 1.5 months corresponds to an HR = 0.8.

the ESMO Magnitude of Clinical Benefit Scale (ESMO-MCBS v1.0)

The ESMO Magnitude of Clinical Benefit Scale version 1 (ESMO-MCBS v1.0) (Appendix 1) has been developed only for solid cancers. Given the profound differences between the curative and palliative settings, the tool is presented in two parts. Form 1 is used to evaluate adjuvant and other treatments with curative intent. Form 2 (a, b or c) is used to evaluate non-curative interventions, with form 2a for studies with OS as the primary outcome, form 2b for studies with PFS or TTP as primary outcomes, 2c for studies with QoL, toxicity or response rate (RR) as primary outcomes and for non-inferiority studies. Form 2a is prognostically sub-stratified for studies where the control arm produced OS greater or less than or equal to 1 year and form 2b for studies where the control arm produced PFS greater or less than or equal to 6 months.

eligibility for application of the ESMO-MCBS

The ESMO-MCBS can be applied to comparative outcome studies evaluating the relative benefit of treatments using outcomes of survival, QoL, surrogate outcomes for survival or QoL (DFI, EFS, TTR, PFS and TTP) or treatment toxicity in solid cancers. Eligible studies can have either a randomised or comparative cohort design [35, 36] or a meta-analysis which report statistically significant benefit from any one, or more of the evaluated outcomes. When more than one study has evaluated a single clinical question, results derived from well-powered registration trials should be given priority.

Studies with pre-planned subgroup analyses with a maximum of three subgroups can be scored. When statistically significant results are reported for more than one subgroup, then each of these should be evaluated separately. Subgroups not showing statistically significant results are not graded. Except for studies that incorporate collection of tissue samples to enable re-stratification based on new genetic or other biomarkers, findings from un-planned (post hoc) subgroup analysis cannot be graded and they can only be used as foundation for hypothesis generation.

form 1

This form is used for adjuvant and neoadjuvant therapies and for localised or metastatic diseases being treated with curative intent. This scale is graded A, B or C. Grades A and B represent a high level of clinical benefit (Figure 3). The scale makes allowance for early data demonstrating high DFS without mature survival data. Studies initially evaluated based on DFS criteria alone will need to be revaluated when mature survival data is available. Hyper-mature data from studies that were un-blinded after compelling early results with subsequent access to the superior arm are contaminated, subsequently late intention-to-treat (ITT) follow-up data are not evaluable [37, 38]. Pathological complete remission from neoadjuvant therapies is not included as a criteria for clinical benefit because of lack of consistent evidence that it is a valid surrogate for survival in clinical studies [39–42].

Figure 3.

Visualisation of ESMO-MCB scores for curative and non-curative setting. A & B and 5 and 4 represent the grades with substantial improvement.

Figure 1.

Use of threshold HR in the ESMO-MCBS exemplified for HR threshold of 0.65.

Figure 2.

The correspondence between an HR value and the minimum absolute gain in months considered as beneficial according to the ESMO-MCBS by median survival (OS or PFS) for control.

forms 2

These forms are used for studies of new agents or approaches in the management of cancers without curative intent. This scale is graded 5, 4, 3, 2, 1, where grades 5 and 4 represent a high level of proven clinical benefit (Figure 3).

form 2a

This version is used for therapies evaluated using a primary outcome of OS. The form is stratified by median OS of the control arm ≤12 and >12 months. Preliminary grading takes into consideration HR and median survival gain as well as late survival advantage and is reported on a 4-point scale. When there is differential grading between the median and late survival gain, the higher score prevails. Preliminary scores can be upgraded by 1 point when the experimental arm demonstrates improved QoL or delayed deterioration in QoL using a validated scale or substantial reduction in grade 3 or 4 toxicity. A score of 5 can only be achieved when optimal survival outcomes are further enhanced by data indicating reduced toxicity or improved QoL.

form 2b

This version is used for therapies evaluated using a primary end point of PFS or TTP. The form is stratified by median duration of PFS of the control arm ≤6 and >6 months. The maximal preliminary score is discounted to 3 because PFS and TTP are surrogate outcomes with a less reliable relationship to improved survival or QoL [18, 20–23]. In studies that allow crossover on subsequent therapy, this may be the best available evidence of activity since subsequent therapies may reduce the likelihood of observing survival benefit.

Preliminary scores derived from PFS studies can be upgraded or downgraded depending on secondary outcomes such as toxicity data, improvement in OS or data derived from QoL evaluation. This form incorporates an adverse effect criterion for down-grading in cases of severe toxicity compared with the control arm. If an OS advantage is observed as a secondary outcome, scores are upgraded using the scale on form 2a. In PFS studies that evaluate global QoL, positive findings (as evidenced by statistically significant improvement in global QoL or delayed deterioration in QoL) will upgrade the evaluation by 1 point and, in the absence of survival advantage, the absence of QoL advantage will result in a down-grading by 1 point.

form 2c

This form is used for therapies evaluated in non-inferiority (equivalence) studies and for studies in which the primary outcomes are QoL, toxicity or RR.

field testing of ESMO-MCBS

ESMO-MCBS has been applied in a wide range of solid tumours by members of the ESMO-MCBS Task Force, the ESMO Guidelines Committee and a range of invited experts (Tables 3–12).

Table 3.

Field testing ESMO-MCBS v1.0: lung cancer

Lung cancer
Medication (new versus control)	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Erlotinib versus carboplatin gemcitabine	OPTIMAL, CTONG-0802	First-line stage IIIb or IV non-squamous, with EGFR mutation	PFS	4.6 months	8.5 months	0.16 (0.10–0.26)					12% less serious  adverse events	4	[43]
Erlotinib versus platinum-based chemotherapy doublet	EURTAC	First-line stage IIIb or IV non-squamous, with EGFR mutation	PFS (crossover allowed)	5.2 months	4.5 months	0.37 (0.25−0.54)	19.5 months		NS		15% less severe  adverse  reactions	4	[44]
Gefitinib versus carboplatin + paclitaxel	IPASS	First-line stage IIIb or IV adenocarcinoma, with EGFR  mutation	PFS (crossover allowed)	6.3 months	3.3 months	0.48 (0.34–0.67)				Improved	Reduced toxicity	4	[45, 46]
Afatinib versus Cisplatin + pemetrexed	LUX—Lung 3	First-line stage IIIb or IV adenocarcinoma with EGFR mutation (Del19/L858R)	PFS (crossover allowed)	6.9 months 6.9 months	4.2 months 6.7 months	0.58 (0.43–0.78) 0.47 (0.34–0.65)				Improved Improved		4	[47, 48]
Crizotinib versus chemotherapy		First-line stage IIIb or IV non-squamous, with ALK mutation	PFS (crossover allowed)	3.0 months	4.7 months	0.49 (0.37–0.64)				Improved	1% increased  toxic death	4	[49]
Crizotinib versus cisplatin + pemetrexed		First-line stage IIIb or IV non-squamous, with ALK mutation	PFS	7.0 months	3.9 months	0.45 (0.35–0.60)				Improved		4	[50]
Pemetrexed versus placebo		Stage IIIb or IV disease maintenance after responding to four cycles platinum doublet	PFS stratified for histology (non-squamous)	2.6 months	1.9 months	0.47 (0.37–0.60)	10.3 months	5.2 months	0.70 (0.56–0.88)			4	[51]
Cisplatin pemetrexed versus cisplatin gemcitabine		First-line stage IIIb or IV (non-squamous)	OS (non-inferiority)				10.4 months	1.4 months	0.81 (0.70–0.94)		Less grade 3 + toxicity neutropenia anaemia thrombocytopenia	4	[52]
Chemotherapy ±  palliative care		Stage IV  non-small-cell  ECOG <2	QoL				8.9 months	2.7 months	HR for death in control arm 1.7 (1.14–2.54)	Improved		4	[53]
Paclitaxel/carboplatin ± bevacizumab		First-line stage IIIb or IV, non-squamous	OS				10.3 months	2.0 months	0.79 (0.67–0.92)			2	[54]
Erlotinib versus placebo	SATURN	Stage IIIb or IV disease maintenance after responding to four to six cycles platinum doublet	PFS	11.1 weeks	1.2 weeks	0.71 (0.62–0.82)	11.0 months	1.0 months	0.81 (0.70–0.95)			1	[55]

Table 4.

Field testing ESMO-MCBS v1.0: breast cancer

Breast cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Chemotherapy ± trastuzumab	HERA	(Neo)adjuvant HER-2-positive tumours	DFS	2-year DFS 77.4%	8.40%	0.54 (0.43–0.67)						A	[56]
T-DM1 versus lapatinib + capecitabine	EMILIA	Second-line metastatic after trastuzumab failure	PFS and OS	6.4 months	3.2 months	0.65 (0.55–0.77)	25 months	6.8 months	0.68 (0.55–0.85)	Delayed deterioration		5	[57, 58]
Trastuzumab + chemotherapy ± pertuzumab	CLEOPATRA	First-line metastatic	PFS	12.4 months	6 months	0.62 (0.52–0.84)	40.8 months	15.7 months	0.68 (0.56−0.84)	No improvement		4	[59–62]
Lapatinib ± trastuzumab	EGF104900	Third-line metastatic	PFS	2 months	1 months	0.73 (0.57–0.93)	9.5 months	4.5 months	0.74 (0.57–0.97)			4	[63, 64]
Capecitabine ± lapatinib		Second-line metastatic after trastuzumab failure	PFS	4.4 months	4 months	0.49 (0.34–0.71)			NS			3	[65]
Eribulin versus other chemotherapy	EMBRACE	Third-line metastatic after anthracycline and taxane	OS				10.6 months	2.5 months	0.81 (0.66–0.99)			2	[66]
Paclitaxel ± bevacizumab		First-line metastatic	PFS	5.9 months	5.8 months	0.60 (0.51–0.70)			NS	No improvement		2	[24]
Exemestane ± everolimus	BOLERO-2	Metastatic after failure of aromatase inhibitor (with PFS >6 months)	PFS	4.1 months	6.5 months	0.43 (0.35–0.54)			NS	No improvement		2	[67]

Table 5.

Field testing ESMO-MCBS v1.0: prostate cancer

Prostate cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Best standard non-chemotherapy or radiotherapy treatment ± radium-223	ALSYMPCA	Castration refractory and bone pain	OS				11.3 months	3.6 months	0.70 (0.55–0.88)	Improved		5	[68]
Prednisone ± abiraterone		Castration refractory after docetaxel	OS				10.9 months	3.9 months	0.65 (0.54–0.77)			4	[69]
Enzalutamide versus placebo	AFFIRM	Castration refractory after docetaxel	OS				13.6 months	4.8 months	0.63 (0.53–0.75)	Improved		4	[70]
Enzalutamide versus placebo	PREVAIL	Castration refractory pre-docetaxel	PFS and OS	3.2 months	>12 months	0.19 (0.15–0.23)	30.2 months	2.2 months	0.71 (0.60–0.84)	Improved		3	[71]
Docetaxel(Q7 or Q21) prednisone versus mitoxantrone + prednisone		Castration refractory	OS				16.5 months	2.4 months (Q21) 0.9 months (Q7)	0.76 (0.62–0.94) 0.83 (0.70–0.99)	Improved Improved		3	[72]
Cabazitaxel + prednisone versus mitoxantrone + prednisone	TROPIC	Castration refractory after docetaxel	OS				12.7 months	2.4 months	0.70 (0.59–0.83)			2	[73]

Table 6.

Field testing ESMO-MCBS v1.0: colorectal cancer

Colorectal cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
FOLFOX4 ± panitumumab	PRIME	First-line metastatic (post hoc KRAS, NRAS BRAF WT)	PFS	7.9 months	2.3 months	0.72 (0.58–0.90)	20.2 months	5.8 months	0.78 (0.62–0.99)			4	[74]
Panitumumab + mFOLFOX6 versus bevacizumab + mFOLFOX6	PEAK	First-line metastatic (KRAS-WT)	PFS			NS	24.3 months	9.9 months	0.62 (0.44–0.89)			4a	[75]
FOLFIRI ± cetuximab	CRYSTAL	First-line metastatic stratified for KRAS-WT (post hoc KRAS, NRAS WT)	PFS	8.4 months	3.0 months	0.56 (0.41–0.76)	20.2 months	8.2 months	0.69 (0.54–0.88)			4	[76]
Cetuximab versus best supportive care		Refractory metastatic KRAS-WT	OS	1.9 months	1.8 months	0.4 (0.30–0.54)	4.8 months	4.7 months	0.55 (0.41–0.740			4	[77]
FOLFOX4 ± panitumumab	PRIME	First-line metastatic KRAS-WT	PFS	8 months	1.6 months	0.80 (0.66–0.97)	19.4 months	4.4 months	0.83 (0.70–0.98)			3	[78, 79]
FOLFIRI ± cetuximab	CRYSTAL	First-line metastatic stratified for KRAS-WT	PFS	8.4 months	1.5 months	0.70 (0.56–0.87)	20 months	3.5 months	0.80 (0.67–0.95)			3	[80, 81]
ILF ± bevacizumab		First-line metastatic	OS				15.6 months	4.7 months	0.66 (0.54–0.81)			3	[82]
FOLFIRI ± panitumumab		Second-line metastatic KRAS-WT	PFS	3.9 months	2 months	0.73 (0.59–0.90)						3	[83]
FOLFOX ± bevacizumab versus bevacizumab alone	E3200	Second-line metastatic after FOLFIRI	OS				10.8 months	2.1 months	0.75 (0.63–0.89)			2	[84]
Panitumumab, versus best supportive care		Third-line metastatic stratified for KRAS	PFS	7.3 weeks	5 weeks	0.45 (0.34–0.59)						2	[85]
FOLFIRI bevacizumab versus FOLFOXIRI bevacizumab		First-line metastatic	PFS	9.7 months	2.4 months	0.75 (0.62–0.90)			NS			2	[86]
TAS-102 versus placebo	CONCOURSE	Third-line or beyond metastatic	OS				5.3 months	1.8 months	0.68 (0.058–0.81)			2	[87]
Regorafenib versus placebo	CORRECT	Third-line metastatic	OS				5 months	1.4 months	077 (0.64–0.94)			1	[88]
Second-line chemotherapy ± bevacizumab	ML18147	Second line beyond progression on bevacizumab	OS				9.6 months	1.5 months	0.81 (0.69–0.94)			1	[89]
FOLFIRI ± aflibercept	VELOUR	Second line after oxaliplatin-based treatment	OS	4.7 months	2.2 months	0.76 (0.66–0.87)	12.1 months	1.5 months	0.82 (0.71–0.94)			1	[90]
FOLFIRI ± Ramucirumab	RAISE	Second-line metastatic after bevacizumab, oxaliplatin, fluoropyrimidine	OS				11.7 months	1.6 months	0.84 (0.73–0.97)			1	[91]

^aUnbalanced crossover.

Table 7.

Field testing ESMO-MCBS v1.0: ovarian cancer

Ovarian cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Paclitaxel or topotecanor liposomal doxorubicin ± bevacizumab	AURELIA	Recurrent platinum resistant	PFS (crossover allowed)	3.4 months	3.3 months	0.48 (0.38–0.60)				Improved		4	[92, 93]
Paclitaxel and carboplatin (five or six cycles) ± bevacizumab till 18 cycles or progression	ICON7	High-risk, early-stage post-resection or advanced ovarian or primary peritoneal	PFS stratified for stage and risk of progression	(All) 22.4 months (high risk) 14.5 months	1.7 months 3.6 months	0.81 (0.70–0.94) 0.73 (0.60–0.90)	28.8 months	7.8 months	NS 0.64 (0.48–0.85)			1 4	[94]
Gemcitabine and carboplatin ± bevacizumab	OCEANS	Recurrent platinum sensitive	PFS (crossover allowed)	8.4 months	4 months	0.48 (0.39–0.61)						3	[95]
Paclitaxel and carboplatin (6 cycles) ± bevacizumab continual till 10 months or progression	GOG 218	Incompletely resected stage III and stage IV	PFS (crossover allowed)	10.3 months	Bevacizumab  continual 3.9 months	0.72 (0.63–0.82)			NS			3	[96]
Liposomal doxorubicin ± trabectedin	OVA-301	Second-line metastatic	PFS stratified for platinum sensitivity/ resistance	(sensitive) 7.5 months (resistant) 5.8 months	1.7 months 1.5 months	0.73 (0.56–0.95) 0.79 (0.65–0.96)						2	[97]
Olaparib versus placebo		BRCA ovarian cancer in remission	PFS	4.3 months	6.9 months	0.18 (0.10–0.31)			NS	Not improved		2	[98]

Table 8.

Field testing ESMO-MCBS v1.0: renal cell cancer

Renal cell cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Pazopanib versus sunitinib	COMPARZ	First-line metastatic RCC with clear-cell component	PFS non-inferiority	9.5 months		NS					Reduced	4	[99]
Temsirolimus versus interferon versus combined		First-line poor-prognosis metastatic RCC	OS				7.3 months	(TEM alone) 3.3 months	0.73 (0.58–0.92)			4	[100]
Sunitinib versus interferon		First-line metastatic	PFS crossover allowed	5 months	6 months	0.42 (0.32–0.054)	21.8 months	4.6 months	NS	Improved		4	[101, 102]
Axitinib versus sorafenib	AXIS	Previously treated metastatic RCC	PFS	4.7 months	2.0 months	0.66 (0.55–0.81)						3	[103]
Sorafenib versus placebo	TARGET	Second line locally advanced or metastatic	OS	2.8 months	2.7 months	0.44 (0.35–0.55)	15.9 months	3.4 months	0.77 (0.63–0.95)			3	[104]
Everolimus versus placebo	RECORD1	Second or third line after TKI metastatic RCC	PFS crossover allowed	1.9 months	2.1 months	0.30 (0.22–0.40)						3	[105]
Pazopanib versus placebo		Second line locally advanced or metastatic	PFS crossover allowed	4.2 months	5 months	0.46 (0.34–0.62)						3	[106]
Interferon ± bevacizumab	AVOREN	First-line metastatic RCC with clear cell	PFS	5.4 months	4.6 months	0.63 (0.52–0.75)						3	[107]
Interferon ± bevacizumab	CALGB 90206	First-line metastatic RCC with clear cell	OS amended to PFS	5.2 months	3.3 months	0.71 (0.66–0.83)						1	[108]

Table 9.

Field testing ESMO-MCBS v1.0: Sarcoma

Sarcoma
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Imatinib 1 year versus placebo	ACOSOG Z9001	Adjuvant for GIST	RFS stratified for risk	1-year RFS 83%	13%	0.35 (0.22–0.53)						A	[109]
3 versus 1 year imatinib	SSG XVIII	Adjuvant for high-risk GIST	5-year RFS	48%	18%	0.46 (0.32–0.65)						A	[110]
Sunitinib versus placebo		Advanced GIST second line after imatinib	TTP crossover allowed	6.4 weeks	16.9 weeks	0.33 (0.23–0.47)						3	[111]
Regorafenib versus placebo	GRID	Third line after imatinib and sunitinib	PFS crossover allowed	0.9 months	3.7 months	0.27 (0.19–0.39)						3	[112]
Pazopanib versus placebo	PALETTE	Previously treated non-GIST metastatic soft tissue sarcoma	PFS	1.6 months	3 months	0.31 (0.24–0.40)						3	[113]
Ridaforolimus versus placebo	SUCCEED	Sarcoma after response or stable disease with first-line treatment	PFS	14.6 weeks	3.1 weeks	0.72 (0.61–0.85)						1	[114]

Table 10.

Field testing ESMO-MCBS v1.0: melanoma

Melanoma
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
Ipilimumab ± glycoprotein 100 vaccine versus vaccine alone		Previously treated metastatic	OS				6.4 months	3.7 months	0.69 (0.56–0.85)			4	[115]
Vemurafenib versus dacarbazine	BRIM-3	First line or second line after IL-2 metastatic with BRAF V600E mutation	PFS and OS	1.6 months	4.7 months	0.26 (0.20–0.33)	9.7 months	3.9 months	0.70 (0.57–0.87)			4	[116, 117]
Trametinib versus dacarbazine or paclitaxel	METRIC	Unresectable or metastatic with BRAF V600E mutation	PFS (crossover allowed)	1.5 months	3.3 months	0.45 (0.33–0.63)	6 months: 67%	14%		Improved		4a	[118, 119]
Dabrafenib ± trametinib		First line unresectable or metastatic with BRAF V600E mutation	Toxicity, PFS	5.8 months	3.6 months	0.30 (0.25–0.62)					12% reduction skin cancer	4	[120]
Dabrafenib versus dacarbazine		First line unresectable or metastatic with BRAF V600E mutation	PFS (crossover allowed)	2.7 months	2.1 months	0.30 (0.18–0.51)				Improved		4	[121, 122]
Dabrafenib + trametinib versus vemurafenib		First line unresectable or metastatic with BRAF V600E mutation	OS	7.3 months	4.1 months	0.69 (0.53–0.89)	1 year: 65%	7%	0.69 (0.53–0.89)		17% reduction skin cancer	4*	[123]
Vemurafenib ± cobimetinib		First line unresectable or metastatic with BRAF V600E mutation	PFS	6.2 months	3.7 months	0.51 (0.39–0.68)	9 months: 73%	8%			9% reduction skin cancer	4*	[124]
Dacarbazine ± nivolumab		First line unresectable or metastatic BRAF V600-WT	OS	2.2 months	2.9 months	0.43 (0.34–0.56)	10.8 months	6+ months	0.42 (0.25–0.73)			4*	[125]
Dacarbazine ± ipilimumab		First-line metastatic	OS (crossover allowed)				3-year survival 12.2% 9.1 months	8.60% 2.1 months	0.33 (0.24–0.53			3	[126, 127]

^aImmature survival data.

Table 11.

Field testing ESMO-MCBS v1.0: pancreatic cancer

Pancreatic cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0-MCBS	Ref.
FOLFIRINOX versus gemcitabine		First line advanced or metastatic,good PS	OS (crossover allowed)				6.8 months	4.4 months	0.57 (0.45–0.73)	Delayed deterioration		5	[128]
Gemcitabine ± nab-paclitaxel		First line advanced or metastatic,good PS	OS				6.7 months	1.8 months	0.72 (0.61–0.83)  5% gain at 24 months			3	[129]
Gemcitabine ± erlotinib		First line advanced or metastatic	OS				5.9 months	0.3 months	0.82 (0.69–0.99)			1	[130]

Table 12.

Field testing ESMO-MCBS v1.0: gastro-oesophageal cancer

Gastro-oesophageal cancer
Medication	Trial name	Setting	Primary outcome	PFS control	PFS gain	PFS HR	OS control	OS gain	OS HR	QoL	Toxicity	ESM0- MCBS	Ref.
Surgery ± perioperative epirubicin, cisplatin, 5-FU	ISRCTN 93793971	Gastric or distal oesophagus stage II–III	OS				5 years: 23%	13%	0.66 (0.53–0.81)			A	[131]
Surgery ± perioperative cisplatin/5-FU		Gastric or distal oesophagus stage II–III	OS				5 years: 24%	14%	0.69 (0.50–0.95)			A	[132]
Ramucirumab versus placebo	REGARD	Second-line gastro-oesophageal or gastric cancer after cisplatin/5-FU	OS				3.2 months	2 months	0.78 (0.60–0.99)			2	[133]

When discrepancies between graders were observed, this was generally related to either inaccurate data extraction, variable interpretation of the significance and severity of toxicity data, or errors in applying the data to the correct grading criteria.

discussion

inherent challenges in developing standard Clinical Benefit Scale

The substantial variability of study designs (crossover, non-crossover and partial crossover), planned outcomes and reported outcomes inherently challenge the process of developing a unified scale of clinical benefit. This challenge is all the greater in an era in which both researchers and regulatory authorities are employing surrogate outcome indicators as primary end points for both research and registration criteria [5]. A unified scaling approach requires a process of relative weighting of evidence that demands conceptual rigor, careful reviews of the validity and strength of surrogate end points and clinical nuance.

validity of the ESMO-MCBS

The ESMO-MCBS version 1 (ESMO-MCBS v1.0) provides an objective and reproducible approach that allows comparisons of the magnitude of benefit between studies that incorporate different primary outcomes (OS, PFS, QoL) and different designs through a process of variable weighting of primary outcomes and adjustments for significant secondary outcomes and toxicity.

The development process has been compliant with the criteria for ‘accountability for reasonableness’ which represent the ethical gold-standard for a fair priority setting process in public policy [134, 135]. The validity of the ESMO-MCBS is derived from (i) clinically relevant and reasonable criteria for prioritisation of different types of benefit, i.e. that cure takes precedence over deferral of death, direct end points such as survival and QoL take precedence over less reliable surrogates such as PFS or RR and that the interpretation of the evidence for benefit derived from indirect primary outcomes (such as PFS or RR) may be influenced by secondary outcome data, (ii) coherence: procedural agreements regarding the evidence to be used/not used, how it will be analysed and evaluated, and precautions to minimising bias (including conflict of interest issues) based upon an understanding of the relative strengths and weaknesses of the usual measured outcomes OS and QoL, and their surrogates [13–23, 136] and rigorous bio statistical review, (iii) wide applicability over a range of solid cancers and a range of prognoses that have been rigorously tested (iv) statistical validity and (v) a transparent process of development with scope for peer review, appeal and revision.

ESMO-MCBS scores for a specific therapy are not generalisable to indications outside the confines of the context in which they have been evaluated. Consequently, the ESMO-MCBS score for a particular medication or therapeutic approach may vary depending on the specifics of the indication and may vary between studies.

limitations of the ESMO-MCBS v1.0

The ESMO-MCBS can only be applied to comparative research outcomes; it is therefore not applicable when evidence of benefit derives from single-arm studies. This limits its utility in the uncommon situation in which registration is granted on the basis of outcomes reported from single-arm studies.

The process of relative weighting of evidence and the thresholds for HR and absolute gains involves judgements and subjective considerations which are amenable to dispute and challenge and indeed, this is invited as part of the dynamic process of peer review and further development.

factors that may skew or alter ESMO-MCBS scores

control arm evaluation

The ESMO-MCBS evaluates data derived from comparative research, either randomised phase II [137] or phase III studies or cohort studies. The validity of the results may be influenced by the quality and design of the study. Design issues are critical insofar as studies that incorporate a relatively weak control arm may generate the impression of exaggerated benefit. This was manifest in studies evaluating treatment options for hormone refractory prostate cancer where one study used mitoxantrone/prednisone as the control arm [73] based on the findings of a phase III study comparing prednisone versus the combination of prednisone and mitoxantrone which demonstrated improved QoL but no survival advantage for the combination therapy [138] and others used prednisone alone [69] or placebo [70].

crossover

Crossover, or subsequent treatment of control arm patients with biologically similar agent, severely compromises the ability to derive reliable data regarding the survival advantage of treatments in phase III studies. This factor may impact on OS results as illustrated by the study of dacarbazine versus ipilimumab in metastatic melanoma [126] in which the evidence for survival advantage was diluted by the crossover provision in the study. In some instances in which strong PFS advantage is seen, crossover of this type will obscure the potential survival benefit of the new treatment. Statistical approaches to estimate longer term clinical outcomes despite substantial treatment crossover have been developed [139, 140], and applied [141–144]. While these approaches are encouraging, they incorporate a range of assumptions and are not universally accepted [145].

unbalanced crossover

In other instances, unbalanced crossover may exaggerate differences in survival. For instance, in the PEAK study comparing FOLFOX6 with either bevacizumab or panitumumab among the patients with KRAS wild-type tumours, only 38% of those in the bevacizumab arm received any EGFR antibody in subsequent therapy [146]. Although this study showed a survival advantage of 9.9 months over a baseline of 24.3 months for patient initiated on treatment with panitumumab, it remains unclear as to whether this was affected by the sequence of treatments or if it was the result that more than half of the patients in the bevacizumab arm were never exposed to an EGFR antibody.

follow-up reports

In some studies, first reports are followed up with subsequent further relevant data analysis. This is particularly true when mature survival data were not available in studies with a primary outcome of PFS or DFS and in studies that have incorporated post hoc stratification based on refined appreciation of tumour biology that may impact on outcome evaluation.

Both of these phenomena were observed in the three publications reporting the findings from the same study on FOLFOX4 ± panitumumab for the first-line treatment of KRAS wild-type colorectal cancer [74, 78, 79]. The study, which did allow for crossover to other EGFR antibodies, had PFS as a primary end point. The initial publication demonstrated a modest PFS advantage with non-significant median OS gain [78]. The subsequent publication of mature data demonstrated a significant OS gain [79] with the greatest benefit restricted to patients with KRAS, NRAS, BRAF wild-type tumours [74]. Almost identical data maturation was observed in the CRYSTAL study evaluating FOLFIRI± cetuxumab in the same clinical setting [76, 80, 81].

Maturation of survival data also increased the ESMO-MCBS score of vemurafenib in the treatment of metastatic melanoma [116, 117] from ESMO-MCBS 3 based on PFS to 4, based on OS.

using data from the ESMO-MCBS

The ESMO-MCBS incorporates a structured, rational and valid approach to data interpretation and analysis that reduces the tendency to have judgements affected by bias or uninformed and/or idiosyncratic data interpretation. Consequently, application of the scale reduces the likelihood that statements of clinical benefit will be distorted by either overestimation or overstatement on one extreme or, nihilism at the other. This structured and disciplined approach to deriving estimates of clinically meaningful benefit from published data can be used in a range of settings.

public policy applications

Grading derived from the ESMO-MCBS provides a backbone for value evaluations for cancer medicines. Medicines and therapies that fall into the ESMO-MCBS A + B for curative therapies and 4 + 5 for non-curative therapies should be emphasized for accelerated assessment of value and cost-effectiveness. While a high ESMO-MCBS score does not automatically imply high value (that depends on the price), the scale can be utilised by to frame such considerations [147] and can help public policy-makers advance ‘accountability for reasonableness’ in resource allocation deliberations [134, 135].

formulation of clinical guidelines

The prevailing current practice of the National Comprehensive Cancer Network (NCCN), American Society of Clinical Oncology (ASCO), ESMO and the National Cancer Institute (NCI) in their guidelines is to grade the ‘level of evidence’ supporting the efficacy of therapeutic interventions; grading the evidence as very high when derived from meta-analyses of well-conducted phase III studies, or from large well-conducted phase III studies relative to lower levels such as that derived from non-randomised studies, anecdote or expert clinical opinion. A major shortcoming of this approach is that it may result in a high level of evidence irrespective of the actual magnitude of the benefit observed, even if the magnitude of the benefit is very limited [148]. This discrepancy has been emphasized by the Grading of Recommendations Assessment, Development and Evaluation (GRADE) Working Group which was formed in 2000 to improve the quality of guideline formulation. The GRADE working group emphasised that a particular quality of evidence does not necessarily imply a particular strength of recommendation [149, 150]. They have developed and championed a widely endorsed approach emphasising appropriate framing of research and guideline questions [151], evaluation of the strength of recommendations that incorporates evaluation of the balance between desirable and undesirable outcomes (estimated effects), and the confidence in the magnitude effect of the interventions on important outcomes [152].

This recommendation can be accomplished by describing both the level of benefit and the level of evidence for recommended therapeutic interventions. For cancer therapies, the ESMO-MCBS scale provides a clear, well-structured and validated mechanism to indicate the magnitude of benefit in addition to the level of evidence that can inform both national and international (e.g. ESMO) guidelines.

clinical decision making

The data encapsulated in ESMO-MCBS scoring can help clinicians to weigh the relative merits of competing relevant therapeutic options in situations in which there is no direct comparative data comparing the available therapeutic options. The grading may also be of benefit in explaining the relative merit of therapeutic options to patients and their families. This information may be especially helpful when treatments incorporate substantial out of pocket costs and the real ‘value’ of the treatment needs to be candidly addressed to avoid over investment or sacrifice of limited financial resources to pay for treatments that have only limited magnitude of benefit.

editorial decisions and commentaries

The ESMO-MCBS may be of use to editors, peer reviewers and commentators in considering the clinical significance of research findings from randomised clinical studies, cohort studies and meta-analyses with statistically significant positive findings.

relevance to the ASCO initiatives

ASCO has undertaken two initiatives to help promote the value in cancer care. The first was a working group to propose new thresholds for the approval of cancer medications [153]. For each of four conditions (metastatic colon cancer, metastatic breast cancer, non-small-cell lung cancer and pancreatic cancer), they have proposed thresholds for meaningful clinical benefit improvement defined in terms of minimal increases in OS (absolute and HR) and also thresholds for minimal increases in surrogate indicators including 1-year survival and PFS. Interestingly, in non-curative therapies, the ASCO recommended thresholds for survival benefit correlate very closely to the thresholds for ESMO-MCBS score of 4–5 (in form 2a) and the recommended thresholds for PFS correlate closely with the thresholds for ESMO-MCBS score of 3–4 which is the highest attainable when the primary outcome is PFS (in form 2b). Secondly, ASCO has developed a Value in Cancer Care Task Force that has been charged with the challenge of developing a framework for evaluating value in oncology. While concurring with ESMO that the evaluation of net clinical benefit is key element in the evaluation of value, ASCO has not yet described their proposed approach to evaluate the magnitude of clinical benefit. A key challenge for the future will be to establish whether there can be harmonisation between the different approaches to value in Europe and the United States.

conclusion

ESMO is committed to promoting rational, responsible and affordable cancer care, the importance of organisational integrity, and the promotion of best use of limited health care resources. The ESMO-MCBS v1.0 was born out of these considerations. It represents a first version of a well-validated tool to stratify the magnitude of clinical benefit for new anti-cancer treatments and is applicable over a full range of solid tumours. Based on the data derived from well-structured phase III clinical trials or meta-analyses, the tool uses a rational, structured and consistent approach to derive a relative ranking of the magnitude of benefit that can be anticipated from any new treatment. The ESMO-MCBS is an important first step to the major ongoing task of evaluating value in cancer care which is essential for appropriate uses of limited public and personal resources for affordable cancer care. The ESMO-MCBS will be a dynamic tool and its criteria will be revised on a regular basis pending peer reviewed feedback and developments in cancer research and therapies.

funding

This project was funded by ESMO.

disclosure

The authors have declared the following: UD: lecture fees—Amgen. EGEV: research grants from Roche/Genentech, Amgen, Novartis, Pieris and Servier to the institute, data monitoring committee Biomarin, advisory board Synthon. MJP: board member—PharmaMar; Consultant (honoraria)—Amgen, Astellas, AstraZeneca, Eli Lilly, GSK, Invivis, MSD, Novartis, Pfizer, Roche/Genentech, Sanofi Aventis, Symphogen, Synthon, Verastem; Research grants to Institute: most companies; Speakers bureau/stock ownership: none. AS: advisory Board and Symposia Satellite with; Amgen, Bayer, Celgene, Merck, Roche and Sanofi. CZ: advisory Boards—Roche, Celgene, Bristol Myers-Squibb; lecture fees—Amgen, Bristol Myers-Squibb. All remaining authors have declared no conflicts of interest.

acknowledgements

The authors wish to acknowledge the support and contribution of the ESMO Executive Board, the ESMO Faculty, Members of the ESMO Guidelines Committee, and the logistic and organisational support provided by ESMO Staff and in particular Nicola Latino. Appendix 2 lists biostatistics and oncology colleagues who provided feedback on earlier versions of the scale and the manuscript.

appendix 1

ESMO magnitude of Clinical Benefit Scale v1.0

appendix 2

Fabrice Andre, France; Dirk Arnold, Germany; Paolo A. Ascierto, Italy; Stefan Bielack, Germany; Jean Yves Blay, France; Federico Cappuzzo, Italy; Fatima Cardoso, Portugal; Andrés Cervantes, Spain; Fortunato Ciardiello, Italy; Alan Stuart Coates, Australia; Karina Dahl Steffensen, Denmark; Theo M. De Reijke, The Netherlands; Jean Yves Douillard, France; Reinhard Dummer, Switzerland; Tim Eisen, UK; Enriqueta Felip, Spain; Constantine Gatsonis, USA; Heikki Joensuu, Finland; Ian Judson, UK; Vesa Kataja, Finland; Roberto Labianca, Italy; Jonathan A. Ledermann, UK; Sumithra J. Mandrekar, USA; Stefan Michiels, France; Mansoor Raza Mirza, Denmark; Mustafa Özgüroğlu, Turkey; Chris Parker, UK; Camillo Porta, Italy; Noemi Reguart, Spain; Daniel J. Sargent, USA; Elżbieta Senkus, Poland; Cristiana Sessa, Switzerland; Kirsten Sundby Hall, Norway; JosepTabernero, Spain; Dongsheng Tu, Canada; Johan Vansteenkiste, Belgium.