A recent explosion of interest in comparative effectiveness research (CER) has been accompanied by diverse attempts to define CER and specify CER research methods. We explore how CER is relevant across the cancer control continuum, including prevention, screening, diagnosis, treatment, and survivorship. We review cancer CER research funded by the National Cancer Institute by analyzing project characteristics along the dimensions of cancer type, stage of the cancer continuum, position on the T0 to T4 translational continuum, and the size and representativeness of both the settings and populations studied. We also provide an assessment of cost and resources considerations in CER. One hundred three relevant projects on CER were funded by the National Cancer Institute’s Division of Cancer Control and Population Science between 2009 and 2011. Prevention studies were most frequent (38.8%), and survivorship grants were least frequent (13.5%). Many projects included economic (35.0%) or simulation modeling (10.7%) approaches as well as multilevel behavioral (53.4%) and/or organizational change (54.4%) interventions. Most studies used convenience sampling (54.3%) and studied two or less settings (50.0%). Cancer CER is active and diverse but could be enhanced by a greater focus on knowledge integration, context, relevance to stakeholders, transparency, and population impact.

Processes for generating and evaluating evidence in cancer and other areas of health care are often slow, costly, or too unrepresentative to provide useful evidence to decision makers ( 1 ). Given cancer burden, increasing concerns about cancer treatment costs, and projections about the anticipated number of cancer survivors, there is a clear need for cancer comparative effectiveness research (CER) ( 2 ). Recently, CER has been recommended as a practical approach to determining what works in health care ( 3 ). As defined by the Institute of Medicine, CER is “the generation and synthesis of evidence that compares the benefits and harms of alternative methods to prevent, diagnose, treat, and monitor a clinical condition or to improve the delivery of care” ( 3 ).

Concerns about the effectiveness of health care have promoted interest in CER, culminating in the recently established Patient Centered Outcomes Research Institute (PCORI). PCORI responds to concerns that patients, providers, and caregivers do not have the types of comparative information they need to make choices aligned with their desired health outcomes. Thus, the concepts of personalization and patient-centeredness are embedded in the PCORI ( 4–6 ). For example, one of the tenets of this research includes answers to these questions: “Given my personal characteristics, conditions, and preferences, what should I expect will happen to me?” and “What are my options, and what are the benefits and harms of those options?” ( 7 ). These same questions apply to personalized or precision medicine that seeks to integrate “-omic” information into health care ( 7 ). Personalizing health care according to patient preferences will require incorporating social and behavioral information from the outset, not just when a new application is ready for clinical use. There has been an outpouring of recent articles on CER, including special issues of JAMA (April 18, 2012) and the Journal of Clinical Oncology (October 15, 2012) that included both conceptual papers and a series of large-scale observational studies, three of which addressed cancer. Given the vast amount of emerging information, it remains to be seen how much evidence will have to come from comparative randomized clinical trials, observational studies, natural experiments, adaptive trials, pragmatic intervention studies, and evidence synthesis and modeling, all of which are tools of CER ( 7 ).

Regardless, three things are clear: 1) CER research should be relevant and broadly applicable; 2) it should include information on costs, resources required, and efficiency issues; and 3) it should address public health impact ( 8 , 9 ).

Patient-Centered CER Requires a Strong Role for Public Health

The practice of medicine occurs at multiple levels, including patient–practitioner dyads, health-care organizations, families, communities, and state and federal agencies, all appropriately viewed by the Institute of Medicine as part of the “public health system” ( 10 ). With advances in information technology and a strong consumer empowerment movement, health-care systems and public health organizations will have an increasing role in collecting population level data, developing policies for both informing and protecting consumers, and assuring that the most vulnerable segments of the population benefit from CER knowledge ( 11 ).

As shown in Figure 1 , there are four overlapping, interrelated, and nonlinear phases of research in advancing from basic discoveries to reducing the burden of cancer in populations. This diagram illustrates the highly iterative nature of the scientific discovery to translation cycle. The process starts with the identification of a problem, opportunity, or approach to a health issue (T0). The first research phase (T1) is the more traditional bench-to-bedside model, at the end of which new therapeutics, tests, or other interventions are developed (these could also be policy or behavioral interventions). The second research phase (T2) involves analysis and investigation of whether the interventions improve cancer health outcomes (in randomized trials or other study designs). The end result of T2 is evidence-based guidelines and recommendations by professional organizations and independent panels. T3 research includes investigations to increase uptake and implementation of evidence-based recommendations into cancer practice and prevention programs, and T4 research involves evaluation of effectiveness and cost-effectiveness of such interventions in the real world and diverse populations. Some discoveries move rapidly through this cycle, or skip steps and become adopted without evidence-based recommendations. Many others never progress beyond T1 or T2 ( 12 ).

Figure 1.

Knowledge integration process ( 13 ). This figure depicts the recursive process of integrating knowledge across different types of research so that they inform each other and move toward population health impact in an integrated fashion. Copyright permission from Sheridan Press, American Journal of Public Health .

Figure 1.

Knowledge integration process ( 13 ). This figure depicts the recursive process of integrating knowledge across different types of research so that they inform each other and move toward population health impact in an integrated fashion. Copyright permission from Sheridan Press, American Journal of Public Health .

This translational framework fits very well with concepts of CER, in T2 to T4 phases. T2-related CER involves comparing the efficacy of different interventions in randomized controlled trials and other designs, T3-related CER involves comparing approaches to implementation and dissemination in practice, and T4-related CER involves comparing effectiveness and population outcomes of various dissemination approaches.

A robust “knowledge integration” (center of Figure 1 ) is needed to drive the translational cycle. It involves three closely related, iterative components—knowledge management, synthesis, and translation—and includes stakeholder engagement in all components ( 11 ). Knowledge management involves an ongoing process of obtaining, organizing, and displaying evolving evidence. Knowledge synthesis involves conducting systematic reviews using mixed methods including meta-analysis, realist review, decision analysis, and simulation modeling that combine information from basic, clinical, and population research. Knowledge translation refers to brokering of the knowledge to influence policy, practice, and research. This model has been recently elaborated on in the context of genomic medicine and implementation science ( 5 , 11 , 13 ). We apply it to help evaluate and understand the types of cancer CER that have been funded.

The aims of this article are to explain the rationale for CER as applied to cancer, to summarize the types of cancer CER research funded by the National Cancer Institute (NCI), to reflect on how CER research can be best employed to impact population health in the context of the T0 to T4 framework, and to recommend types of research methods that can produce relevant and translatable CER results.

Summary of Recent CER Funded by the NCI

The Research, Condition, and Disease Categorization (RCDC) process of the National Institutes of Health (NIH) was used to generate a preliminary list of 107 grants coded as CER that were funded through the Division of Cancer Control and Population Sciences of the NCI in fiscal years 2009 to 2011. RCDC is a computerized process used by the NIH to identify grants funded in 233 categories of disease, condition, or research area for the purposes of tracking amounts of NIH funding ( http://report.nih.gov/rcdc/ ). Because these grants were selected based on an automated algorithm, abstracts from these grants were then reviewed by a single investigator (V.P. Doria-Rose) to verify that the grants were indeed CER. A second reviewer (M.L. Brown) then reviewed abstracts from grants that were considered either clearly not CER or ambiguous. Based on consensus of the two reviewers, four grants were excluded (three focused on informatics efforts related to electronic medical records that might be applied to CER efforts but were not in themselves CER, and one focused on racial/ethnic disparities in cancer outcomes). A total of 103 research grants were included; career development (K series) and other types of awards were not examined in this analysis. Grants were classified according to several categories, including type of cancer/prevention focus, points on cancer control continuum, place on T2 to T4 continuum, content area, and methods used ( Table 1 ).

Table 1.

Number and percentage of National Cancer Institute comparative effectiveness research grants with different characteristics

Topic or issue Categories Number of grants* Percentage of grants 
Point(s) on cancer continuum
Total No. of grants: 103
Total No. of categories: 111  
   
 Prevention 40 38.8% 
 Screening 24 23.3% 
 Diagnosis 2.9% 
 Treatment 19 18.4% 
 Survivorship 14 13.5% 
 Multiple 10 9.7% 
 Not stated 1.0% 
Type(s) of cancer
No. of grants: 103
No. of categories: 136  
   
 Breast 23 22.3% 
 Colorectal 18 17.5% 
 Lung 5.8% 
 Prostate 13 12.6% 
 Gynecologic 1.0% 
  Cervical 4.9% 
  Ovarian 1.0% 
 Not applicable 3.9% 
 Multiple 14 13.6% 
 Other (melanoma, glioblastoma, esophageal, bladder, etc) 6.8% 
Prevention focused    
 Tobacco 32 30.1% 
 Dietary 3.9% 
 Physical activity 3.9% 
 Obesity 3.9% 
 Other (energy balance) 1.0% 
Place on T2–T4 continuum    
 T2 → recommendations or policies 7.8% 
 T3 → cancer care/implementation 54 52.4% 
 T4 → population cancer impact 29 28.2% 
Content area    
 Drug 13 12.6% 
 Device 1.0% 
 Behavioral 55 53.4% 
 Organizational system 56 54.4% 
 Policy 5.8% 
 Methods 6930 5.8% 
 Multiple  8.7% 
 Other (test = 17; surgery = 7; radiation = 5; other = 1)  29.1% 
Methods used    
 Experimental, RCT 59 57.3% 
 Experimental, other 1.0% 
 Observational 26 25.3% 
 Modeling 11 10.7% 
 Cost/economic 36 35.0% 
 Infrastructure 12 101 11.7% 
 Methods development  9.7% 
 Other  1.0% 
Topic or issue Categories Number of grants* Percentage of grants 
Point(s) on cancer continuum
Total No. of grants: 103
Total No. of categories: 111  
   
 Prevention 40 38.8% 
 Screening 24 23.3% 
 Diagnosis 2.9% 
 Treatment 19 18.4% 
 Survivorship 14 13.5% 
 Multiple 10 9.7% 
 Not stated 1.0% 
Type(s) of cancer
No. of grants: 103
No. of categories: 136  
   
 Breast 23 22.3% 
 Colorectal 18 17.5% 
 Lung 5.8% 
 Prostate 13 12.6% 
 Gynecologic 1.0% 
  Cervical 4.9% 
  Ovarian 1.0% 
 Not applicable 3.9% 
 Multiple 14 13.6% 
 Other (melanoma, glioblastoma, esophageal, bladder, etc) 6.8% 
Prevention focused    
 Tobacco 32 30.1% 
 Dietary 3.9% 
 Physical activity 3.9% 
 Obesity 3.9% 
 Other (energy balance) 1.0% 
Place on T2–T4 continuum    
 T2 → recommendations or policies 7.8% 
 T3 → cancer care/implementation 54 52.4% 
 T4 → population cancer impact 29 28.2% 
Content area    
 Drug 13 12.6% 
 Device 1.0% 
 Behavioral 55 53.4% 
 Organizational system 56 54.4% 
 Policy 5.8% 
 Methods 6930 5.8% 
 Multiple  8.7% 
 Other (test = 17; surgery = 7; radiation = 5; other = 1)  29.1% 
Methods used    
 Experimental, RCT 59 57.3% 
 Experimental, other 1.0% 
 Observational 26 25.3% 
 Modeling 11 10.7% 
 Cost/economic 36 35.0% 
 Infrastructure 12 101 11.7% 
 Methods development  9.7% 
 Other  1.0% 

* Numbers add up to more than 103 in some cases because a given grant fit into more than one category. RCT = randomized controlled trial.

In addition, we reviewed the full grant proposals to assess information related to the samples and settings studied and the costs and resources involved. These additional analyses were conducted for 94 funded studies for which the entire grant was available in an electronic form. The following criteria for both patient/consumer sample and contextual setting were abstracted: the type of setting (eg, cancer center, clinical setting, workplace, school, community) and the number and recruitment procedures used for both settings and individual participants (ie, convenience sample or no information; some effort to include diverse sample; systematic effort to ensure representativeness). Two factors related to costs were recorded: 1) if there was any report of costs being considered in the design of the study and 2) if non-monetary costs were reported. Finally, three variables were coded related to the transparency of the grant application: 1) if justification was provided for the inclusion and exclusion of settings, 2) if justification was provided for inclusion and exclusion of individual participants, and 3) if there was any discussion of study limitations or alternative approaches.

Types of CER Currently Funded

A wide range of cancer CER research has been funded by the NCI ( Table 1 ). The vast majority of included grants (92%) were the result of investigator-initiated applications; only eight grants were funded through specific requests for applications or program announcements. The 103 projects were broadly distributed across types of cancers/risk factors, points on the translational continuum, and methods. There were more prevention grants (38.8%) and screening projects (23.3%) than treatment grants and survivorship grants (13.5%; palliative and end-of-life care grants were especially infrequent), and diagnosis studies were funded least often (2.9%). Breast, colorectal, and prostate cancers were all well represented, with 13.6% of the grants studying multiple types of cancer. Among the prevention grants, tobacco studies dominated; there were few grants in other prevention areas.

Funded projects evaluated several types of interventions, the most common being behavioral (53.4%) and organizational/systems approaches (54.4%); these two categories dominated the content areas studied ( Table 1 ). In contrast, only six policy grants were identified, and only two focused predominantly or exclusively on policy issues.

A variety of research methods were proposed, with randomized trials, cost/economic studies, and observational studies all well represented. One area underemphasized was innovative, alternative experimental design methods: all but one of the 60 experimental grants funded were randomized controlled trials. Several grants (10.7%) proposed using simulation modeling or cost/economic analyses (35.0% of the projects). In terms of the grants location on the “translation highway”, T1 and likely some T2 grants were excluded because of our focus on cancer control (rather than discovery, which is generally not funded by the Division of Cancer Control and Population Sciences), but there were many grants at both T3 (52.4%) and T4 (28.2%) points on the continuum.

Although there were numerous examples of T3 and T4 research conducted in relevant real-world settings, such as clinical (37.5%) and community (26.0%) ( Table 2 ) settings, few of these reported comprehensively on recruitment at either the setting or patient level, and very few studied an entire region or population-based sample. In contrast, less than 5% of the funded grants focused on worksite or school settings, and very few specifically reported selecting low-resource settings. A majority of grants proposed to report on economic and resources-required issues (50.5%–58.9%). There was variability in the sample sizes studied, with the most common sample sizes being either greater than 1000 (40.4%) or less than 100 (24.6%). In contrast, 50.0% of the grants proposed to study only one or two settings, and only 3.2% included a large number of settings. Sampling strategies at both the setting and patient level were predominantly convenience sampling (54.3%), with only 12.8% using any systematic or representative sampling methods. In terms of transparency, a minority of projects included any justification or discussion of their setting sampling methods or exclusions, and only 28.4% included discussion of limitations or alternative approaches.

Table 2.

Setting, sampling issues, representativeness, and resources reporting proposed in cancer comparative effectiveness research crants

Issue and categories within topic Percentage of grants* 
Setting  
 Cancer center 11.5% 
 Clinical 37.5% 
 Worksite 4.2% 
 School 
 Community 26.0% 
 Virtual 6.3% 
 Other 14.6% 
Sampling strategies  
 Convenience 54.3% 
 Some diversity 33.0% 
 Systematic/representative 12.8% 
Sample size  
 Of settings 50% 
  1 or 2 40.4% 
  3–7 6.4% 
  8–15 3.2% 
  Large number of settings  
 Of patients  
  <100 24.6% 
  100–499 17.6% 
  500–599 8.8% 
  600–999 8.8% 
  ≥1000 40.4% 
  Missing/not specified 39 grants 
Resources reporting  
 Considered in design/intervention 58.9% 
 Report nonmonetary resources required 50.5% 
Transparency issues  
 Justify setting sampling and exclusions 45.3% 
 Justify participant sampling and exclusions 61.1% 
 Report on limitations 28.4% 
Issue and categories within topic Percentage of grants* 
Setting  
 Cancer center 11.5% 
 Clinical 37.5% 
 Worksite 4.2% 
 School 
 Community 26.0% 
 Virtual 6.3% 
 Other 14.6% 
Sampling strategies  
 Convenience 54.3% 
 Some diversity 33.0% 
 Systematic/representative 12.8% 
Sample size  
 Of settings 50% 
  1 or 2 40.4% 
  3–7 6.4% 
  8–15 3.2% 
  Large number of settings  
 Of patients  
  <100 24.6% 
  100–499 17.6% 
  500–599 8.8% 
  600–999 8.8% 
  ≥1000 40.4% 
  Missing/not specified 39 grants 
Resources reporting  
 Considered in design/intervention 58.9% 
 Report nonmonetary resources required 50.5% 
Transparency issues  
 Justify setting sampling and exclusions 45.3% 
 Justify participant sampling and exclusions 61.1% 
 Report on limitations 28.4% 

* Percentages add up to greater than 100% in some cases because a given grant fit into more than one category.

Reflections and Discussion

Our review assessed 103 cancer control CER studies funded across a variety of cancer types, stages of the cancer care continuum, and translational phases of research. These studies should provide a wealth of information for decision makers, including clinicians and patient/families, in the near future. These studies extend well beyond a narrow definition of CER as studying only drugs and devices and have included a large number of studies on systems/organizational interventions and behavioral interventions. Many grants focused on cost and economic issues, and several proposed simulation modeling, an approach capable of producing rapid results and potentially alerting scientists and policy makers to both positive and negative interactions and unintended consequences ( 11 , 14 , 15 ).

Successful implementation of cancer prevention and care and translation of CER require partnership with and full participation of patients and families, health-care staff, and organizational decision makers, as well as community stakeholders ( 16–19 ). Hood and Friend ( 6 ) have viewed this participatory role as important in terms of creating the necessary information and information technology to deal with the exponential growth of genomic and biological data on individuals. It also sets the stage for stakeholder engagement, equity, access, cost and coverage, and choice among alternative approaches ( 20 , 21 ).

Although there were many T3 and T4 studies, only a minority made efforts to ensure that their settings and samples studies were representative. When health-care resources are limited and inequitably distributed (eg, millions of people have no or inadequate health care coverage), there is an ethical obligation to assure that the national investment in research leads to tangible health benefits for all and does not worsen existing health disparities.

Recommended Types of Rapid, Flexible, Contextual CER

The need for rapid, relevant evidence that is actionable by patients and families, clinical teams, and policy makers implies that different and expanded types of research methods will be needed to accomplish the aims of cancer CER ( 5 , 8 ). The recent report of the PCORI Methodology Committee outlines many of the relevant issues ( http://www.pcori.org/assets/Preliminary-Draft-Methodology-Report.pdf ). Below, we summarize the characteristics of the types of research methods needed by this new field of “CER-T,” or CER that will translate into practice and policy ( 8 ).

A key characteristic of CER is that it is practical or pragmatic ( 22–24 ) and provides information for making real-world decisions. This is why CER requires comparisons that involve real-world options rather than placebo or no treatment. Building on the work of Gierisch et al. ( 25 ), we recommend that CER investigators consider inclusion of “minimal interventions needed for change” comparison conditions, in much the same way that evaluations of new cardio-protective medications might include comparisons with aspirin. New and expensive resource-intensive interventions should be demonstrably more effective than standard care or a minimal interventions needed for change comparison before they are recommended for broad adoption (Glasgow RE, Fisher L, Strycker LA, et al, unpublished data).

Other features of pragmatic designs are that they address issues important to stakeholders and that they employ multiple outcomes relevant to these stakeholders ( 22 , 26 ). We encourage cancer CER grantees to address the CONSORT pragmatic trials criteria when reporting their results ( 23 ), and the related pragmatic explanatory continuum indicator survey figure ( 27 ) can help to increase transparency in reporting results and research methods. Many of these issues, especially those related to unanticipated outcomes or unintended consequences, are best addressed using mixed methods ( 28 ) or qualitative assessments ( 29 ) ( http://obssr.od.nih.gov/scientific_areas/methodology/mixed_methods_ research/index.aspx ). Measures in a given CER study should address the issues central to that topic but should include among these outcomes both patient-centered measures and broad outcomes such as quality of life ( 13 , 30 ) that allow comparisons across disease and content areas. For example, in the context of genomic medicine, CER offers a variety of methods that can address stakeholders’ needs and help ensure translation of genomic discoveries into population health benefits ( 31 ).

One of the greatest opportunities to make cancer research CER-T ( 8 ) is consistent, comprehensive, and transparent reporting of costs of interventions and resources required and outcomes such as cost-effectiveness and cost–benefit ( 32 , 33 ), especially because comparisons such as cost per quality adjusted life year are explicitly excluded from PCORI ( 34 , 35 ). We recommend that cancer CER research include assessments such as costs of implementing a program and replication costs. Our understanding is that PCORI is not allowed to sponsor any research to produce standard cost-effectiveness analysis results, especially in the form of incremental cost per incremental quality-adjusted life years ( 34 , 35 ). However, because PCORI is mandated to sponsor and carry out research on outcomes of importance to patients and on factors that influence the effectiveness of health care, topics such as health-care cost as a burden on patients and health-care cost and financial arrangements such as coinsurance and copayments can be studied because they affect access and adherence.

To be relevant and actionable, cancer CER-T studies ( 8 ) should report results in ways that are understandable and transparent to local health-care organizations, patients/families, and practitioners ( Table 3 ). This includes standardized reporting of the levels of participation, engagement, and results by key population subgroups. Examples of key subgroups are those related to long-standing health inequities defined by race, ethnicity, education, income, and class and also emerging disparities issues related to factors such as age, number of comorbid conditions, health literacy and numeracy, and place ( 36–39 ). The projects reviewed generally studied only a small number of settings and did not describe their sampling methods or rationale for selecting these settings. A fuller, transparent accounting of results, including the settings, clinicians, and patients invited to participate, and comparisons between those who participate vs those who decline and are considered ineligible are needed to help determine generalizability and applicability to various settings ( 40–42 ).

Table 3.

Methods recommendations for comparative effectiveness research–translate that will rapidly translate and be relevant

Characteristic Translational purpose of this feature 
Pragmatic or practical * To answer questions from patients, practitioners, and policymakers to inform real-world decision making 
Comparison condition(s) are real alternatives; should include minimal interventions needed for change comparators To address practical questions in context of currently available (and usually less expensive) alternatives 
Collects costs and economic data To provide information on resources needed to adopt and replicate in different settings 
Assesses multiple outcomes using mixed methods To provide results that recognize the different priorities of multiple audiences(eg, behavior change, quality of life/functioning, health-care use, impact on health disparities, unintended consequences) 
Uses flexible and multiple research designs to fit question To consider and address key threats to internal and external validity and to assess adaptation and evolution over time 
Evaluates multilevel participation and representativeness  To determine breadth of applicability and assess participation rate and representativeness of participants, settings, staff, and subgroups 
Modeling and longitudinal evaluation methods to capture adaptation‡ To track evolution of programs, policies, and implementation, including changes over time and adjustments made 
Characteristic Translational purpose of this feature 
Pragmatic or practical * To answer questions from patients, practitioners, and policymakers to inform real-world decision making 
Comparison condition(s) are real alternatives; should include minimal interventions needed for change comparators To address practical questions in context of currently available (and usually less expensive) alternatives 
Collects costs and economic data To provide information on resources needed to adopt and replicate in different settings 
Assesses multiple outcomes using mixed methods To provide results that recognize the different priorities of multiple audiences(eg, behavior change, quality of life/functioning, health-care use, impact on health disparities, unintended consequences) 
Uses flexible and multiple research designs to fit question To consider and address key threats to internal and external validity and to assess adaptation and evolution over time 
Evaluates multilevel participation and representativeness  To determine breadth of applicability and assess participation rate and representativeness of participants, settings, staff, and subgroups 
Modeling and longitudinal evaluation methods to capture adaptation‡ To track evolution of programs, policies, and implementation, including changes over time and adjustments made 

* Thorpe et al. ( 27 ).

Keyserling et al. ( 63 ).

Des Jarlais et al. ( 64 ).

Another key issue is reporting of context ( 43 ). Because most CER examines how different treatment or prevention approaches work in the real world, they are context dependent. Interventions often have different effects in different settings and patient populations and are affected by differences in the policy environment, community, health-care system, practice, research team, target audience and actual participants, and local cultural factors ( 20 , 21 , 44–46 ).

Among other tools, simulation modeling can be useful at each stage of the T0 to T4 cycle, especially in examining progression along the cycle in ways that are not possible with single or experimental studies. Systems models are well suited to the dynamic complexity and context dependence that characterize many CER questions ( 47–49 ). The use of modeling techniques also allows researchers to ask CER-T questions that better reflect the complexity of CER phenomena, considering multiple interacting diseases and risks, the interaction of delivery systems and at risk populations, and policy and environmental contexts ( 47 , 50 ). The use of participatory group modeling approaches may be particularly helpful in engaging the wisdom of patients and other stakeholders in patient-centered CER ( 51–53 ).

We recognize that many are concerned about the limitations of modeling and do not view either those methods or the other recommendations above as a panacea. We do think that such methods have been historically underutilized and deserve consideration as approaches that should be combined with other methodologies, including randomized trials (which are often very expensive, slow, and of limited generalizability) and natural experiments—especially those involving large numbers of representative samples. We do not think that any one method provides perfect answers but that a balance of complementary methods is needed, especially greater use of historically underutilized, less expensive, and innovative methods that produce more rapid and relevant results to produce answers to CER questions ( 6–9 , 11 ).

Finally, almost no experimental or quasi-experimental designs were reported in our review besides randomized controlled trials. We recommend greater emphasis on replication and the convergence of results across different research methods ( 54 ). Multimethod research approaches can be helpful when methods are carefully chosen to complement each other ( 29 , 55 ). This complementarity can involve designs and data approaches, sources, and transdisciplinary teams ( 50 , 56 , 57 ) that have different strengths and limitations. Research using complementary methods can be conducted sequentially or simultaneously ( 58 , 59 ) to triangulate understanding ( 60 ). The combined use of qualitative and quantitative methods ( 61 , 62 ) may be particularly helpful both in making real the “patient-centered” in patient-centered outcomes research and in providing contextual understanding in multilevel research ( 20 ).

This article provides a summary of recently funded NCI research on CER. It is limited in that it is not a comprehensive review of all cancer CER research funded or reported; several other foundations and groups also fund similar research, and it is too early to evaluate the results and impact of these investments. Also we do not have the “denominator” of all the grants that were submitted but not funded to allow us to make comparisons of the content and characteristics of funded vs unfunded grants, which would be of value. Our portfolio analysis was limited in time and scope to grants funded during the period from 2009 to 2011, which provided us a manageable number of grants for review and covers the primary “spike” in CER funding. There are likely a few grants funded before 2009 and will be many more in the future, so this is a snapshot in time, but we feel it is a useful one to identify general features of the field. Finally, one of the reviewers raised the issue of whether, in this time of fiscal austerity, it is feasible to expect grantees to address the issues recommended above given their likely increased costs. Our response is threefold: 1) such actions are needed to achieve the purpose of CER, and there are likely things now being done that could be dropped; 2) we do not expect every grant to include all of the above recommendations, but rather to think more broadly about these issues and how their reports will be used; and finally, 3) many of the recommendations do not demand added time or resources, but simply increased transparency in reporting.

Conclusion

There is an important need for CER knowledge across both the cancer control continuum and the phases of translational research. Recent cancer CER has spread across most of these dimensions. It is encouraging to see a number of different content areas and research methods being used. Greater emphasis will be needed on survivorship and diagnosis research, as well as more innovation on experimental designs.

We especially encourage cancer CER-T as summarized above and in Table 3 . Even more important than multiple types of CER, we need knowledge integration across the T levels and across different experimental methods. A lot is being done at different levels, types, and stages of cancer research, but more effort is needed to integrate resulting knowledge ( 12 ). We hope that in a few years, cancer CER funding will have advanced this type of understanding also.

The authors do not have any conflicts to disclose. The opinions expressed are those of the authors and do not necessarily reflect those of the National Cancer Institute or the Centers for Disease Control and Prevention.

References

1.
Lenfant
C
.
Clinical research to clinical practice—lost in translation?
N Engl J Med
  .
2003
;
349
(
9
):
868
874
.
2.
Parry
C
Kent
EE
Mariotto
AB
Alfano
CM
Rowland
JH
.
Cancer survivors: a booming population
.
Cancer Epidemiol Biomarkers Prev
  .
2011
;
20
(
10
):
1996
2005
.
3.
Iglehart
JK
.
Prioritizing comparative-effectiveness research—IOM recommendations
.
N Eng J Med
  .
2009
;
361
(
4
):
325
328
.
4.
Zerhouni
EA
.
Transforming medicine through discovery. Major trends in biomedical research
.
Bull Acad Natl Med
  .
2007
;
191
(
8
):
1685
1694
.
5.
Khoury
MJ
Gwinn
ML
Glasgow
RE
Kramer
BS
.
A population approach to precision medicine
.
Am J Prev Med
  .
2012
;
42
(
6
):
639
645
.
6.
Hood
L
Friend
SH
.
Predictive, personalized, preventive, participatory (P4) cancer medicine
.
Nat Rev Clin Oncol
  .
2011
;
8
(
3
):
184
187
.
7.
Garber
AM
Tunis
SR
.
Does comparative-effectiveness research threaten personalized medicine?
N Engl J Med
  .
2009
;
360
(
19
):
1925
1927
.
8.
Glasgow
RE
Steiner
JF
.
Comparative effectiveness research to accerlate translation: Recommendations for an emerging field of science
. In:
Brownson
RC
Colditz
G
Proctor
E
, eds.
Dissemination and Implementation Research in Health: Translating Science and Practice
  .
New York
:
Oxford University Press
;
2012
:
72
93
.
9.
Glasgow
RE
Chambers
D
.
Developing robust, sustainable, implementation systems using rigorous, rapid and relevant science
.
Clin Transl Sci
  .
2012
;
5
(
1
):
48
55
.
10.
Institute of Medicine
.
Who Will Keep the Public Healthy?
 
Washington, DC
:
National Academy Press
;
2002
.
11.
Khoury
MJ
Clauser
SB
Freedman
AN
et al
.
Population sciences, translational research, and the opportunities and challenges for genomics to reduce the burden of cancer in the 21st century
.
Cancer Epidemiol Biomarkers Prev
  .
2011
;
20
(
10
):
2105
2114
.
12.
Khoury
MJ
Gwinn
M
Joannidis
JP
.
The emergence of translational epidemiology: from scientific discovery to population health impact
.
Am J Epidemiol
  .
2010
;
172
(
5
):
517
524
.
13.
Glasgow
RE
Vinson
C
Chambers
D
et al
.
National Institutes of Health approaches to dissemination and implementation science: current and future directions
.
Am J Public Health
  .
2012
;
102
(
7
):
1274
1281
.
14.
McGlynn
EA
.
Intended and unintended consequences: what should we really worry about?
Med Care
  .
2007
;
45
(
1
):
3
5
.
15.
Safavi
K
.
Unintended consequences: getting the most while avoiding the worst
.
J Healthc Manag
  .
2006
;
51
(
6
):
355
359
.
16.
Selby
JV
Beal
AC
Frank
L
.
The Patient-Centered Outcomes Research Institute (PCORI) national priorities for research and initial research agenda
.
JAMA
  .
2012
;
307
(
15
):
1583
1584
.
17.
Hebert
JR
Brandt
HM
Armstead
CA
Adams
SA
Steck
SE
.
Interdisciplinary, translational, and community-based participatory research: finding a common language to improve cancer research
.
Cancer Epidemiol Biomarkers Prev
  .
2009
;
18
(
4
):
1213
1217
.
18.
Cargo
M
Mercer
SL
.
The value and challenges of participatory research: strengthening its practice
.
Annu Rev Public Health
  .
2008
;
29
:
325
350
.
19.
Stange
KC
.
The Journal of Participatory Medicine: Setting its sights on community of practice
.
J Participat Med
  . http://www.jopm.org/evidence/reviews/2009/10/21/the-journal-of-participatory-medicine-setting-its-sights-on-a-community-of-practice/ . Accessed October 26, 2012.
20.
Stange
KC
Breslau
ES
Dietrich
AJ
Glasgow
RE
.
State-of-the-art and future directions in multilevel interventions across the cancer control continuum
.
J Natl Cancer Inst Monogr
  .
2012
;
2012
(
44
):
20
31
.
21.
Glasgow
RE
Green
LW
Taylor
MV
Stange
KC
.
An evidence integration triangle for aligning science with policy and practice
.
Am J Prev Med
  .
2012
;
42
(
6
):
646
654
.
22.
Tunis
SR
Stryer
DB
Clancey
CM
.
Practical clinical trials: increasing the value of clinical research for decision making in clinical and health policy
.
JAMA
  .
2003
;
290
:
1624
1632
.
23.
Zwarenstein
M
Treweek
S
Gagnier
JJ
et al
.
Improving the reporting of pragmatic trials: an extension of the CONSORT statement
.
BMJ
  .
2008
;
337
:
a2390
24.
Glasgow
RE
Gaglio
B
Bennett
G
et al
.
Applying the PRECIS criteria to describe three effectiveness trials of weight loss in obese patients with comorbid conditions
.
Health Serv Res
  .
2012
;
47
(
3 Pt 1
):
1051
1067
.
25.
Gierisch
JM
DeFrank
JT
Bowling
JM
et al
.
Finding the minimal intervention needed for sustained mammography adherence
.
Am J Prev Med
  .
2010
;
39
(
4
):
334
344
.
26.
Glasgow
RE
Magid
DJ
Beck
A
Ritzwoller
D
Estabrooks
PA
.
Practical clinical trials for translating research to practice: design and measurement recommendations
.
Med Care
  .
2005
;
43
(
6
):
551
557
.
27.
Thorpe
KE
Zwarenstein
M
Oxman
AD
et al
.
A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers
.
CMAJ
  .
2009
;
180
(
10
):
E47
E57
.
28.
Cresswell
JW
Klassen
AC
Plano Clark
VL
Clegg Smith
K
.
Best Practices for Mixed Models in the Health Sciences
  .
Washington, DC
:
Office of Behavioral and Social Sciences Research
;
2011
.
29.
Crabtree
BF
Miller
WL
.
Doing Qualitative Research
  .
Thousand Oaks, CA
:
Sage Publications
;
1999
.
30.
Kaplan
RM
.
The significance of quality of life in health care
.
Qual Life Res
  .
2003
;
12
(
Suppl 1
):
3
16
.
31.
Goddard
KA
Knaus
WA
Whitlock
E
et al
.
Building the evidence base for decision making in cancer genomic medicine using comparative effectiveness research
.
Genet Med
  .
2012
;
14
(
7
):
633
642
.
32.
Gold
MR
Siegel
JE
Russell
LB
Weinstein
MC
.
Cost-Effectiveness in Health and Medicine
  .
New York
:
Oxford University Press
;
2003
.
33.
Ritzwoller
DP
Sukhanova
A
Gaglio
B
Glasgow
RE
.
Costing behavioral interventions: a practical guide to enhance translation
.
Ann Behav Med
  .
2009
;
37
(
2
):
218
227
.
34.
Neumann
PJ
Weinstein
MC
.
Legislating against use of cost-effectiveness information
.
N Engl J Med
  .
2010
;
363
(
16
):
1495
1497
.
35.
Wilkerson
J
PCORI Head Vows Not To Do Cost-Effectiveness Studies, But Notes Gray Areas. http://insidehealthpolicy com/Inside-Health-General/Public-Content/pcori-head-vows-not-to-do-cost-effectiveness-studies-but- notes-gray-areas/menu-id-869 html . Accessed July 7, 2012.
36.
Mercer
SW
Watt
GC
.
The inverse care law: clinical primary care encounters in deprived and affluent areas of Scotland
.
Ann Fam Med
  .
2007
;
5
(
6
):
503
510
.
37.
Wolff
M
Bates
T
Beck
B
et al
.
Cancer prevention in underserved African American communities: barriers and effective strategies—a review of the literature
.
WMJ
  .
2003
;
102
(
5
):
36
40
.
38.
Wasson
JH
Ahles
T
Johnson
D
et al
.
Resource planning for patient-centered, collaborative care
.
J Ambul Care Manage
  .
2006
;
29
(
3
):
207
214
.
39.
Asch
SM
Kerr
EA
Keesey
J
et al
.
Who is at greatest risk for receiving poor-quality health care?
N Engl J Med
  .
2006
;
354
(
11
):
1147
1156
.
40.
Kessler
RS
Purcell
EP
Glasgow
RE
et al
.
What does it mean to ‘‘employ’’ the RE-AIM model?
Eval Health Prof
  .
2012
;
36
(
1
):
44
46
.
41.
Pawson
R
Greenhalgh
T
Harvey
G
Walshe
K
.
Realist review: a new method of systematic review designed for complex policy interventions
.
J Health Serv Res Policy
  .
2005
;
10
(
S1
):
S21
S39
.
42.
Glasgow
RE
Strycker
LA
Kurz
D
et al
.
Recruitment for an internet-based diabetes self-management program: scientific and ethical implications
.
Ann Behav Med
  .
2010
;
40
(
1
):
40
48
.
43.
Stange
KC
Glasgow
RE
.
Considering and reporting important contextual factors.
. In: Agency for Health Care Research and Quality, eds.
Methods Brief for the AHRQ Initiative in Patient-Centered Medical Home (PCMH)
  .
Rockville, MD
:
Agency for Healthcare Research and Quality
,
2012
.
44.
Kessler
R
Glasgow
RE
.
A proposal to speed translation of healthcare research into practice: dramatic change is needed
.
Am J Prev Med
  .
2011
;
40
(
6
):
637
644
.
45.
Diez Roux
AV
.
Complex systems thinking and current impasses in health disparities research
.
Am J Public Health
  .
2011
;
101
(
9
):
1627
1634
.
46.
Weiner
SJ
Schwartz
A
Weaver
F
et al
.
Contextual errors and failures in individualizing patient care: a multicenter study
.
Ann Intern Med
  .
2010
;
153
(
2
):
69
75
.
47.
Homer
JB
Hirsch
GB
.
System dynamics modeling for public health: background and opportunities
.
Am J Public Health
  .
2006
;
96
(
3
):
452
458
.
48.
Auchincloss
AH
Diez Roux
AV
.
A new tool for epidemiology: the usefulness of dynamic-agent models in understanding place effects on health
.
Am J Epidemiol
  .
2008
;
168
(
1
):
1
8
.
49.
Gilbert
N
.
Agent-Based Models
  .
London
:
Sage Publications
;
2007
.
50.
Mabry
PL
Olster
DH
Morgan
GD
Abrams
DB
.
Interdisciplinarity and systems science to improve population health: a view from the NIH Office of Behavioral and Social Sciences Research
.
Am J Prev Med
  .
2008
;
35
(
2 Suppl
):
S211
S224
.
51.
Hovmand
PS
Brennan
L
Chalise
N
Whose model is it anyway? Paper presented at the 29th International Conference of the System Dynamics Society
;
July 25–29
,
2011
;
Washington, DC
.
52.
Rose
J
Riolo
R
Hovmand
P
et al
.
Modeling the paradox of primary care.
. In:
Martin
C
Sturmberg
J
, eds.
Handbook on Systems and Complexity in Health
  .
New York
:
Springer Press
;
2012
.
53.
Rouwette
E
Vennix
JAM
Mullekom
TV
.
Group model building effectiveness: a review of assessment studies
.
Syst Dyn Rev
  .
2006
;
18
(
1
):
5
45
.
54.
Huberman
AM
Crandall
DP
.
Fitting words to numbers: Multisite/multimethod research in educational dissemination
.
Am Behav Sci
  .
1982
;
26
(
1
):
62
83
.
55.
Stange
KC
Crabtree
BF
Miller
WL
.
Publishing multimethod research
.
Ann Fam Med
  .
2006
;
4
(
4
):
292
294
.
56.
Stokols
D
.
Toward a science of transdisciplinary action research
.
Am J Community Psychol
  .
2006
;
38
(
1–2
):
63
77
.
57.
Syme
SL
.
The science of team science: assessing the value of transdisciplinary research
.
Am J Prev Med
  .
2008
;
35
(
2 Suppl
):
S94
S95
.
58.
Stange
KC
Miller
WL
Crabtree
BF
O’Connor
PJ
Zyzanski
SJ
.
Multimethod research: approaches for integrating qualitative and quantitative methods
.
J Gen Intern Med
  .
1994
;
9
(
5
):
278
282
.
59.
Louis
KS
.
Multisite/multimethod studies: an introduction
.
Am Behav Sci
  .
1982
;
26
(
1
):
6
22
.
60.
Chesla
CA
.
When qualitative and quantitative findings do not converge
.
West J Nurs Res
  .
1992
;
14
(
5
):
681
685
.
61.
Stange
KC
Zyzanski
SJ
.
Integrating qualitative and quantitative research methods
.
Fam Med
  .
1989
;
21
(
6
):
448
451
.
62.
Creswell
JW
.
Research Design: Qualitative, Quantitative, and Mixed Methods Approaches
  .
2nd ed
.
Thousand Oaks, CA
:
Sage Publications
;
2003
.
63.
Samuel-Hodge
CD
Garcia
BA
Johnston
LF
et al
.
Rationale, design, and sample characteristics of a practical randomized trial to assess a weight loss intervention for low-income women: the Weight-Wise II Program
.
Contemp Clin Trials
  .
2011
;
33
(
1
):
93
103
.
64.
Des Jarlais
DC
Lyles
C
Crepaz
N
TREND Group
.
Improving the reporting quality of nonrandomized evaluations of behavioral and public health interventions: the TREND statement
.
Am J Public Health
  .
2004
;
94
(
3
):
361
366
.