Trade-off analysis of agri-food systems for sustainable research and development

Abstract

Tradeoff Analysis (TOA) is an approach to positive analysis that combines foresight analysis and simulation modeling tools from the relevant disciplines, including economics, in a participatory process designed to formulate and evaluate forward-looking, strategic decisions under high levels of uncertainty in complex systems. We motivate TOA with a prototype framework for the design and evaluation of public-good agricultural research for sustainable development. We discuss the advantages of TOA over conventional economic analysis-Benefit-Cost Analysis-for the design and evaluation of sustainable development pathways. The remainder of the paper describes the currently available modeling tools and their strengths and limitations for use in TOA, and illustrates recent applications with cross-scale case studies. We conclude with a discussion of the opportunities and challenges for the use of foresight analysis and TOA in research priority setting and management at global and project levels, using the case of the CGIAR to illustrate.

Trade-off analysis, Agri-food system, Agricultural research, Sustainable development

1 Introduction

Sustainability is a guiding concept and goal for our economies and societies, and for the agri-food system (Antle and Ray 2020). The United Nations has established seventeen Sustainable Development Goals (SDGs), including SDG2, Zero Hunger. Many agricultural and other public good-oriented research organizations have embraced this and other SDGs. For example, the Consultative Group for International Agricultural Research (CGIAR)—the consortium of international agricultural research centers—is redefining its mission as ‘ending hunger by 2030—through science to transform food, land and water systems in a climate crisis’ (CGIAR System Reference Group 2019).

There are many calls for what humanity ‘must’ do to achieve SDG2 and similar goals. For example, the widely publicized EAT-Lancet Commission Report (EAT-Lancet Commission 2019) argues:

Agriculture and fisheries must not only produce enough calories to feed a growing global population but must also produce a diversity of foods that nurture human health and support environmental sustainability … The current global food system requires a new agricultural revolution that is based on sustainable intensification and driven by sustainability and system innovation. This would entail at least a 75% reduction of yield gaps on current cropland, radical improvements in fertilizer and water use efficiency, recycling of phosphorus, redistribution of global use of nitrogen and phosphorus, implementing climate mitigation options including changes in crop and feed management, and enhancing biodiversity within agricultural systems. In addition, to achieve negative emissions globally as per the Paris Agreement, the global food system must become a net carbon sink from 2040 and onward. (pp. 22–3; original emphasis)

Like other studies before it, the EAT-Lancet Commission Report calls for changes in agri-food systems and sets ‘science-based’ targets for meeting these normative goals. What these kinds of reports do not do, and cannot do, is to say how the diverse agricultural systems underpinning the equally diverse local to global food systems can change to meet SDG2 and related goals. This is the challenge for national and international research organizations including the CGIAR—to contribute to the global process of creating the science to meet critical agricultural and food system challenges in ways consistent with diverse private and public interests.

Sachs (2015) defines sustainable development as a positive analytical method and a set of normative goals such as the SDGs. As societies strive to achieve SDGs and other goals, the scale, scope, and complexity of agri-food systems and their linkages to natural and human systems will lead to inevitable trade-offs among and between the economic, environmental, and social impacts of these systems. Trade-off analysis (TOA) is an approach to positive analysis that combines simulation modeling tools from the relevant disciplines, including economics, with other analytical tools such as foresight analysis (FA) and monitoring and evaluation (ME) in a participatory process designed to inform a wide array of stakeholders. Thus, compared to the research paradigm of mainstream applied economics that involves using observational (historical) data to test hypotheses about causal effects of one or a small number of factors, TOA is designed to help decision-makers formulate and evaluate forward-looking, strategic decisions under high levels of uncertainty in complex systems. The goal is to help research organizations such as the CGIAR establish evidence-based priorities, assess progress toward goals, and adapt priorities as new information becomes available and uncertainties are resolved. In this paper, we describe TOA for analysis of agri-food systems, but the same logic supports the use of these tools more generally for sustainability research and development.

To motivate the use of TOA, in the second section of this paper we present a prototype framework for the design and evaluation of public good agricultural research for sustainable development. The third section uses an example to describe what we mean by TOA and its advantages over conventional economic analysis—benefit–cost analysis (BCA)—for the design and evaluation of sustainable development pathways. The remainder of the paper describes the currently available modeling tools and their strengths and limitations for use in TOA, and illustrates recent applications with cross-scale case studies. We conclude with a discussion of the opportunities and challenges for the use of TOA in research priority setting and management at global and project levels, using the case of the CGIAR to illustrate.

2 Designing and evaluating agricultural research for sustainable development

Agricultural research organizations around the world are now confronting the challenge of adapting their investment decisions to achieve sustainable development. We use the current efforts by the CGIAR's reform toward ‘One CGIAR’ to illustrate the processes that are likely to be involved (Barret et al. 2020). The CGIAR leadership and its donors have identified five principal sustainable development impact areas associated with its overarching goal of ending hunger by 2030: nutrition and food security; poverty reduction, livelihoods, and jobs; gender equality, youth, and social inclusion; climate adaptation and greenhouse gas (GHG) reduction; and environmental health and biodiversity. As illustrated in Fig. 1, this goal will be pursued through a small set of global and regional programs that address major challenges (CGIAR System Reference Group 2019).

Figure 1.

Proposed use of FA and TOA in CGIAR priority setting, research portfolio design, and monitoring and evaluation.

Source: Antle and Valdivia (2020b).

Figure 1 illustrates how FA and TOA could fit into the CGIAR global and regional priority setting processes. These processes should be based on data and evidence created through the incorporation of FA and TOA and aligned with the theory of change and ME processes at the spatial and temporal scales appropriate to selected impact indicators. The global research strategy is built around stakeholder consultations that identify major global challenges and corresponding impact indicators in the five priority impact areas, including goals for these indicators. For each major challenge area:

• FA with stakeholders and scientists identifies a set of plausible future development pathways and agri-food system innovation scenarios implied by each challenge area.

• The pathways and innovation scenarios are used to implement TOA to evaluate the sets or ranges of impacts that are feasible, using the selected impact indicators. These scenarios should include sufficient detail to enable project design in the following stages of the process.

• Based on the FA and TOA results, consultations with funders and stakeholders lead to the co-creation of a research strategy, including prospective research projects consistent with goals in the five impact areas, and allocation of resources among projects.

• The research strategy is communicated to the regional programs for consultation with partner countries and organizations, leading to regional and national research portfolios of projects consistent with capabilities in the regions.

• Research project implementation proceeds with activities to support project-level FA, TOA, and ME. FA and TOA are implemented at the beginning of a project to inform the design of the initial technology ‘funnel’ consistent with impact goals. Accumulation of data over time and improvement of models allow updating of TOA analysis. This information is passed to project ME and also synthesized with data from other projects and the challenge areas and communicated to the global priority setting process.

• At the project level, a set of protocols is followed to ensure cost-effective implementation of FA, TOA, and ME.

  • Allocate resources to FA, TOA, and ME to ensure they are part of every project and scaled to the project's budget.

  • Identify quantifiable indicators in each impact area; if indicators are not quantifiable, then qualitative indicators should be identified.

  • Identify disciplinary data and models needed to carry out quantitative TOA. Data should meet standards for minimum data to support initial and ongoing TOA and meet standards to be findable, accessible, interpretable, ethical, and reproducible.

• Information is fed back from the project level to focus the technology development funnels and periodically review and update institute-wide priorities.

3 TOA of agri-food systems

We define agri-food system TOA as a participatory process using qualitative and quantitative data and modeling tools to evaluate how technological and institutional innovations can improve system performance using foresight methods and relevant metrics described as impact indicators. This section discusses the motivations and rationale for TOA and illustrates the TOA method with the first application of TOA in CGIAR research, and then discusses the conceptual foundations and key elements of TOA.

3.1 Motivations for TOA

TOA addresses decision problems that arise in the management and improvement of complex, dynamic, multiscale systems with high levels of uncertainty. A key property of complex systems, such as agri-food systems, is that the outputs and outcomes and eventual impacts of these systems are multidimensional and incommensurate. For example, in the case we present in the next section, pesticide use potentially impacts farm households’ incomes and the health of farm family members, as well as the health of other members of the community, food consumers, and the environment. The conventional economic decision framework (BCA) would be applied to a problem like this by attempting to quantify and value the economic impacts (say, on farm household incomes), as well as the impacts on human health, in monetary units. Economists argue that people must make choices that imply trade-offs between monetary and non-market outcomes such as health, and thus imply implicit monetary values to non-market outcomes (Loomis 2005). Yet, even if one accepts monetary valuation of non-market outcomes as valid, researchers rarely if ever have the resources available to account for the differing values within a diverse society, such as the multiple stakeholders that are represented in the research priority setting process described in Fig. 1. Thus, by treating outcomes such as health or environmental risk as a monetary value, and aggregating all such values into ‘net benefits’, researchers are obscuring rather than elucidating the trade-offs that exist among outcomes, as well as the distribution of those outcomes among members of society (Capalbo, Antle, and Seavert 2017).

Another motivation for TOA is the ubiquity and magnitude of multidimensional impacts and the diversity of stakeholder interests, values, and social and political decision-making processes. These considerations belie the use of BCA or the maximization of a ‘social welfare function’ to choose technology or policy options. It is implausible in most cases to value all impacts in monetary terms for BCA; thus, even when some non-market impacts can be monetized, other dimensions that are rarely if ever monetized, such as gender-related impacts, are often ignored (Lentz 2020). Consequently, attempting to ‘optimize’ in a subset of the relevant dimensions will generally result in a ‘sub-optimal’ solution, if a well-defined and valid objective function exists. However, in fact there is no one objective function or social welfare functions to be maximized when there are diverse stakeholders with different values. Thus, in actual decision-making contexts such as illustrated in Fig. 1, stakeholders need information about impacts and their inter-relationships (trade-offs and synergies) to help them choose among alternative development pathways, based on their values and on the decision-making processes that are operating in their social and political settings.

Another key feature of forward-looking analysis is uncertainty. Forward-looking analysis deals with uncertainty by using scenario analysis methods. The goal is not to ‘predict’ the future but rather to identify plausible ranges of outcomes, based on plausible ranges of assumptions about the future. An example is the development of ‘Representative Concentration Pathways’ and ‘Shared Socioeconomic Pathways’ by climate scientists to project future climate impacts using global integrated assessment models. Similarly, potential future agri-food system pathways are being evaluated using scenario methods (Lentz 2020; Valdivia et al. 2015, 2020; Zurek, Hebinck, and Selomane 2020). The two case studies presented in Section 4.3 illustrate the use of scenario methods for TOA.

Thus, TOA is motivated by the goal of elucidating potential impacts associated with actual or potential changes in an agri-food system, so that stakeholders can make informed choices among options using their own valuation among multiple, incommensurate outcomes—i.e. outcomes that can be defined in different units. A critical component of TOA implementation is therefore the selection of the relevant metrics of system performance that are described as impact indicators. In large, complex systems, there are many potential impacts, and thus many possible impact indicators. Given that time and other resources are always limited, a key part of TOA implementation is to select a minimally sufficient set of impact indicators to evaluate system performance consistent with stakeholder priorities. This aspect of TOA implementation is a judgment that must be made given the problem at hand, the research objectives, and the resources available. These elements of TOA are illustrated in the following section.

3.2 TOA example: the Rockefeller pesticide studies

Two of the first uses of TOA in international agricultural research were studies supported by the Rockefeller Foundation in the 1990s, one at the International Rice Research Institute and one at the International Potato Center (CIP). Besides being the first studies of their kind, they laid out the foundations and conceptual framework of TOA and served to illustrate the ‘mechanics’ of TOA. These studies also provided important lessons about the potential unintended consequences of agricultural technology development, and the potential for forward-looking analysis to evaluate options to improve the performance of the production systems in economic, environmental, and health dimensions.

Background: Two studies in the 1990s funded by the Rockefeller Foundation carried out analysis of the economic, environmental, and health impacts of pesticide use in rice production in the Philippines and potato production in Ecuador (Pingali and Roger 1995; Crissman, Antle, and Capalbo 1998). These studies were motivated by the recognition among farm communities, environmental interests, scientists, and the wider society of environmental risks associated with the use of highly toxic pesticides. In additional to environmental contamination, both studies found strong evidence of substantial acute health risks to farm workers and household members, most notably neurological impairment from exposure to highly toxic insecticides and fungicides. Both studies also showed there were trade-offs between farm income and human health with reductions in the use of these pesticides. However, better safety procedures for handling and use, and the use of better management practices such as integrated pest management (IPM) were found to produce ‘win–win’ outcomes, thus effectively improving the terms of the economic–health trade-offs. Development of IPM and farmer education, more effective labeling of pesticides for risk, and restricting use of the most harmful insecticides were among policy recommendations. In both cases, a clear implication for crop breeding was to focus attention on resistance to major disease and insect pests. Notably, in the case of potatoes, development of late blight-resistant potatoes was a priority that also could potentially create win–win economic, environmental, and health outcomes, and was a focus of subsequent research led by CIP.

TOA process: Both studies used a participatory process that brought together scientists and stakeholders (researchers, farmers, local farm organizations, local to national policy and political leaders) to identify key impact indicators—in these studies, the indicators were economic (crop production, farm income), environmental (water contamination from pesticide leaching to ground and runoff to surface water; harm to aquatic life and other species), and human health (acute health effects on farm workers and family members, including impaired neurological function). The indicators in turn were used to identify the economic, environmental, and health data needed, and the disciplinary models needed to model and simulate the systems. Stakeholders (scientists, interest groups, farmers, policymakers) also helped to identify the technology and impact pathways (scenarios) for analysis, including the use of improved self-protection, the adoption of IPM practices, and feasible policy interventions such as pesticide regulation.

Data: A key element of both studies was to identify the data needed to quantify the selected indicators. Detailed farm production data were needed that could be combined with data on the practices and health of farm workers. As a result, a novel feature of both studies was collaboration with medical teams to collect data on pesticide use as well as the health of farm workers making the applications. These workers typically applied highly toxic ‘cocktails’ of multiple pesticides using backpack sprayers with little protection from exposure during mixing and application.

TOA modeling: These studies used simulation models to evaluate the economic and health outcomes associated with changes in pesticide use. Economic outcomes were quantified using econometric models that were simulated over alternative prices and management practices. Equations representing health outcomes were estimated using data collected from medical examinations of farm workers and detailed data on pesticide use (Antle and Pingali 1994; Antle, Cole, and Crissman 1998).

TOA: graphical representation: The simulation models were used to construct ‘trade-off curves’ that show graphically the relationships between two indicators that result from a change in pesticide use, while holding constant the parameters representing the production system technology and the processes generating health and environmental outcomes. For example, in Fig. 2, the ‘base’ or observed potato–pasture production system in Ecuador was simulated to show the relationship between farm income and health risk due to pesticides generated by varying potato prices, while holding other parameters of the model fixed. The changes in economic and health outcomes result from the simulated changes in farmers’ management decisions, including their land allocation between potatoes and other crops, and their use of pesticides and fertilizers. As Fig. 2 shows, at the time of the study, over 50 per cent of the population of farm workers was observed to experience risk of substantial neurological disorders, while generating a per-hectare income of about $1,500.

Figure 2.

Trade-offs between profitability of Ecuadorian potato production and farm worker health. Shaded area represents a ‘no-harm’ set for potential impacts of IPM and safety practices. Safety practices include use of protective clothing and appropriate handling and application. Source: Based on data from Crissman, Antle, and Capalbo (1998).

The high risk of experiencing neurological disorders was due to mixing and application of hazardous fungicides and insecticides using backpack sprayers with minimal or no personal protection and generally unsafe handling practices (e.g. mixing with bare hands; application in dense foliage with ordinary clothing). Even at low potato prices and much lower farm incomes, reductions in the use of fungicides and insecticides were not enough to improve health outcomes substantially.

TOA modeling is used to explore how improvements in system performance could improve outcomes. In this study, two improvements in the system were explored: one was the adoption of IPM that would increase the efficacy of pesticides and reduce the amounts and frequency of use (in the observed system, pesticides were routinely applied about ten times per season); the other was the use of safe handling practices and protective gear that would reduce exposure. Figure 2 shows that IPM alone would allow farms to maintain their incomes while reducing health risk from about 55 to about 35 per cent (i.e. to improve the proportion that was not at risk from about 45 to 65 per cent). Combining IPM with safe practices was estimated to reduce risk to less than 20 per cent (more than 80 per cent safe). In addition, these changes substantially ‘flattened’ the trade-off between income and safety, meaning that higher incomes could be achieved without substantially increasing risk.

Set-based TOA: Working with stakeholders, TOA researchers can identify the pathways that are considered acceptable. For example, in Fig. 2, the goal of ‘doing no harm’ in either dimension would imply the goal of moving the system from its observed point into the shaded area where some improvement in both economic and health outcomes is achieved (a win–win outcome). This type of ‘set-based’ system design is used in industry to provide flexibility in potential development pathways. In a public policy context, it can be used to recognize that there may be a range of preferences among stakeholders and also a range of outcomes may occur as economic and other conditions vary. It is also important to note that some stakeholders might prefer a win–lose or lose–win outcome—that is, some farmers might prefer to accept higher risk in exchange for higher income, while others might prefer to give up some income in exchange for greater safety.

Impact pathways: The set-based discussion shows that many outcomes are possible in complex systems, and will depend on technology and policy interventions. In part as a result of the research in Ecuador, the government instituted efforts to improve pesticide safety, for example by encouraging farmer field schools to provide training in safe use of chemicals, and in 2010 banned the most hazardous insecticides. High levels of pesticide use in the Philippines and many other countries continues with minimal regulation and remains a serious health problem. Unfortunately, the use of hazardous pesticides is now spreading into Africa as agricultural productivity improves, risking a repeat of the experiences in East Asia and Latin America (Sheahan and Barrett 2014). Efforts are being made to improve awareness of the problem and develop solutions, for example the Integrated Production and Pest Management Program supported by the United Nations Food and Agriculture Organization (Settle and Garba 2011).

3.3 Implications for use of TOA by public good research organizations

The Rockefeller pesticide studies illustrate a number of useful lessons for research organizations aiming to achieve multiple outcomes across the economic, environmental, and social dimensions of sustainable development.

Identifying unintended consequences of technologies: New technologies inevitably have unintended consequences. The pesticide example illustrates how a focus on productivity in ‘Green Revolution’ agricultural research led to production systems that provided much needed growth in calorie availability to feed rapidly growing populations, but were not sustainable in the other dimensions such as environment and human health.

Design of sustainable innovation pathways: The TOA process did not provide precise ‘predictions’ of the future; rather, it demonstrated the potential to improve production system performance through research and policy interventions. These studies illustrate how TOA can help design innovations that move systems toward win–win outcomes.

Identification of impact indicators: These studies demonstrated the importance of a minimally sufficient number of quantifiable impact indicators that matter to stakeholders and that demonstrate the importance of trade-offs and the need for improvements in system performance. For example, focusing on acute health impacts was sufficient to indicate the pathway toward needed improvements in the farming system; it was not necessary to carry out more costly analysis of other potential impacts such as cancer or birth defects.

Importance of new and better data: These studies identified critical new data needed—data on farm workers’ health that could be linked to their pesticide use and exposure—to provide the evidence needed on impacts. Today, this remains true in several impact areas such as nutrition and gender.

Importance of appropriate disciplinary models and analytical methods: A multidisciplinary, model-based approach made it possible to simulate outcomes that could not be observed in the field, and could not be obtained from classical experimental approaches. Models could predict out of sample to show potential pathways to improve system performance. Whereas disciplinary models for some environmental impacts were available (i.e. for pesticide transport in the environment), models for health impacts were not available to the research teams, so statistical models needed to be devised. This lack of disciplinary models remains a challenge for several of the impact areas of concern, notably in nutrition and gender, as discussed in Section 4.4.

Methods to communicate impacts: The pesticide project research teams realized that even two-dimensional graphs such as Fig. 2 were not likely to be effective to communicate results of TOA, so other types of data displays and pictures were used. Since then, many innovations in data analytics and visualization, such as the ones illustrated in Section 4, are available and should be used. Particularly useful for presenting TOA results would be recently developed dashboard tools designed to compare and contrast results from multiple scenarios. These include the AgMIP (Agricultural Model Intercomparison and Improvement Project) Impact Explorer (http://agmip-ie.wenr.wur.nl/) and the Food Security Portal and the Global Foresight for Food and Agriculture Tool (http://tools.foodsecurityportal.org/impacts-alternative-agricultural-investments-version-9).

3.4 Participatory modeling framework for TOA

The early pesticide studies discussed above developed a participatory approach that involved scientists and stakeholders to identify the key impact indicators, and the corresponding disciplinary data and models that were needed to quantify those indicators in an agricultural system simulation (Crissman, Antle, and Capalbo 1998). Since the 1990s, there have been many innovations in methods for participatory modeling, as well as many advances in data and models discussed in the following section of this paper.

An example of a participatory modeling approach is the one developed by the AgMIP (Rosenzweig and Hillel 2015), in collaboration with national and international partner organizations, to assess climate impact and adaptation of agricultural systems (Fig. 3). This figure shows the process from initial evaluation of a system using prior knowledge, stakeholder inputs, and modeling results. A key feature of this approach is that scientists and stakeholders identify the key indicators to be used to evaluate the performance of the agricultural systems of interest, and co-design system adaptations to improve performance. These choices then guide the data and models needed for impact evaluation. Results of model simulations are interpreted and communicated to stakeholders using tools such as the web-based Impacts Explorer.

Figure 3.

AgMIP regional integrated assessment process. Source: Valdivia et al. (2019).

The pesticide example illustrates the importance of identifying impact (or sustainability) indicators that can be used to set goals and to evaluate and compare the performance of systems along alternative development pathways. There are a large number of indicators across the three ‘pillars’ of sustainable development (economic, environmental, and social) that are often used in technology impact assessments and TOA (Tables A1–A3; Antle and Ray 2020). As noted in Section 3.3, identification of a minimally sufficient set of impact indicators is a critical element in TOA to guide coordinated data collection and modeling tool selection for system simulation.

Section 4 discusses the wide range of currently available data and tools to implement TOA and related types of impact assessments, such as climate impact assessment and analysis of adaptation options as portrayed in Fig. 3. The key elements of TOA modeling are portrayed in Fig. 4. The early TOA applications, such as the pesticide studies presented in Section 3.2, correspond to the left-hand side of Fig. 4 labeled as ‘Supply Side’. These studies, and many of those at the farm and eco-regional scales discussed in Section 4, represent the farm production system, the farm household, or eco-regions comprised of a population of farms and farm households. These analyses are usually structured as what economists call ‘price-taking’ farms, meaning that the analysis is done for prices set at a given level to represent conditions defined by the scenarios determined relevant to the analysis (i.e. in the participatory development pathway design processes of Fig. 3). However, more recent studies have begun to link these ‘Supply Side’ analyses to the demand side of markets at subnational, national, or global levels, represented by the ‘Demand Side’ in Fig. 4. Currently, this sort of cross-scale linkage is at the frontier of TOA modeling. It provides a way to expand the scope of indicators in an analysis to the demand side, and to incorporate feedbacks from market processes and price changes to the impacts at the farm or eco-regional scale.

Figure 4.

Model components and linkages in TOA. Source: Valdivia, Antle, and Stoorvogel (2012).

An important insight from research on modeling human systems and their environmental impacts is the importance of the similarities and differences in spatial and temporal scales within and across the various processes involved. These systems (physical, biological, human) are each complex and operate at various scales. Understanding these systems, modeling them, and devising ways to link them through their inputs and outputs, or integrate them into larger, ever more complex systems, is a daunting scientific challenge. Most research to date has linked disciplinary models through input and output protocols. The need to combine models gives rise to the need for coordination across disciplines to identify and obtain the data needed. For example, agronomists doing on-farm trials of new crop varieties need to know what data are needed for economic analysis of the adoption and impact of the technology. These insights are the basis for the recommendations in this paper that data standards should be established for ongoing efficient data collection efforts by projects and programs, linked to the impact indicators that are identified in collaboration with stakeholders.

4 Data and models for TOA

This section reviews the application of modeling tools at multiple scales and their features to conduct TOA of agricultural systems.

4.1 Major modeling approaches

A range of tools and approaches have been developed to evaluate impacts of agricultural technologies and their potential trade-offs, and have been used by national and international research organizations since the early years of the Green Revolution. In this section, we briefly discuss the major modeling approaches. The following section reviews a number of modeling studies that have used these approaches.

Cost–benefit analysis was applied in the 1970s to evaluate financial margins of new agricultural technologies (Herdt 1991; Alston et al. 1995). As sustainability began to be accepted as a valid concept to guide economic development, interest in agricultural sustainability led to new analytical approaches that coupled biophysical and economic data and models to assess the sustainability of agricultural systems. As discussed in Section 3, early TOA coupled economic, environmental, and health models to evaluate agricultural sustainability (Antle and Pingali 1994; Crissman, Antle, and Capalbo 1998; Pingali and Rosegrant 1994). Analytical methods used for quantitative impact assessment and TOA are described below; opportunities and challenges for their use in TOA are described in Table 1.

Table 1.

Opportunities and challenges of modeling approaches for TOA. Adapted from Thornton et al. (2018).

Approach	Opportunities	Challenges
Simulation modeling (spatially explicit)	Efficiently assessing spatiotemporal variability	Complexity and uncertainty can be high, precluding decisions
	Allows comparison across different contexts	Calibration and validation are challenging
	Allows exploration of a wide range of scenarios	High data intensity
Simulation modeling (parsimonious)	Generic and can be applied to any production system	Static analysis
	Produces timely and accurate information to support decision making
	Flexible to include multiple indicators
Mathematical programming/optimization methods	Consideration of multiple system objectives	Data intensive, time consuming, difficulty of eliciting household objectives and representing them appropriately (Thornton et al. 2018)
	Flexibility in defining system's objectives	Difficult to address hypothetical situations, other contexts, or scenarios.
Cost–benefit analysis/economic surplus	Applicable in different contexts	Difficult to capture all benefits and costs (e.g. bank account or insurance aspect of cattle)
	Low data intensity	Difficult to include multiple criteria or system's objectives (e.g. poverty, nutritional outcomes)
	Requires less information than other (i.e. econometric, optimization) models.	Required information on price responsiveness of consumers and producers often not available
	Widely used to estimate impact of agricultural technologies	Difficult to include non-economic outcomes (e.g. poverty, nutrition)
Econometrics	Allows estimating direct impacts at multiple levels (farmer, county, or state)	Limited ability to extrapolate responses outside estimation sample (Antle and Capalbo 2001)
	Allows statistical testing of economic theory	Restrictive assumptions associated with choice of functional form (work on flexible technology representations; Carter 1984)
		Data intensive: requires detailed survey data
Qualitative approaches	Incorporates expert and stakeholder views, often reflective of realities in the field	Difficult to compare across different groups of experts or contexts
	Flexibility to incorporate multiple variables and systems’ objectives	Difficulty in relating expert-based scores to measurable variables
	Various existing examples in Climate Smart Agriculture (CSA) research	There can be considerable variation across experts or communities
	Many methods exist, with varying degrees of complexity and ease of implementation	Subject to bias if groups are dominated by certain individuals (e.g. women left out) or if stakeholders deliberately mislead organizers (i.e. tell organizers ‘what they want to hear’)
	Linkable to other approaches (e.g. modeling)
Meta-analysis/ systematic review	Can include multiple sources of potentially disparate (e.g. experimental, model-based) evidence, seeking consensus among these	Difficult to draw generalized conclusions or reach consensus when context specificity is high or evidence is limited
	Can combine multiple indicators into aggregated dimensions, hence useful for CSA	Time consuming if the systematic review is too long and complex (many variables, many studies)
	Systematic review can include adoption rates of practices and factor this into analysis	Difficult to draw conclusions on underlying processes
Spatial analysis/GIS/remote sensing	Allows delineation of target zones or recommendation domains	Dependent on good spatial datasets
	Simplicity	Often difficult to include socioeconomic aspects at high resolution
		Difficult to incorporate systems dynamics, or to assess mixed systems
Integrated assessment modeling	Allows integration of a suite of different models to evaluate synergies and trade-offs	Complex and skill and time consuming to carry out
	Can provide outputs in several dimensions relating to land use, commodity prices, and environmental and health impacts, for example	Conceptual difficulty of model validation and calibration
		Uncertainty bounds on model outputs are often unknown; when known (e.g. Nelson et al. 2014), they may be very large

Simulation modeling: This approach is used to explore options that are not observed or can be difficult to test in reality. These models have been applied to assess agricultural production at various levels of research, including crop, livestock, and farming system`(Klapwijk et al. 2014; Thornton et al. 2018). Simulation models are typically complex and are data intensive (e.g. spatially explicit models). Reduced-form or parsimonious simulation models have been developed as alternative to provide timely information and at low cost.

Mathematical programming and optimization methods: These models are based on linear or quadratic programming to achieve an objective such as input minimization or output maximization, subject to suitable constraints, at the farm or household level. Multicriteria analysis has been used to assess trade-offs of new interventions (Howitt 1995).

Econometric models: Econometric models involve statistical methods using historical datasets on system responses (e.g. crop yields, dairy production, output and input prices). Early applications focused at single crop production functions, usually estimated from experimental data. As econometric methods evolved, multi-crop production analysis was possible using farm survey data. Econometric models have been used with biophysical simulation models to assess trade-offs of production systems (Antle and Capalbo 2001).

Qualitative approaches: Strategic design using scenarios and expert judgment have become widely used tools in FA by the military, business, government, and research. Observational and modeled data can be combined with qualitative judgment to formulate ‘fuzzy’ estimates of future outcomes and trade-offs. A combination of qualitative and quantitative methods is now used in most FA, such as the development of the Shared Socioeconomic Pathways used in climate impact assessments.

Integrated assessment models (IAMs): These models are used to assess current and future agricultural systems under different socioeconomic development pathways, technological change, and climate conditions. IAMs couple disciplinary models (e.g. biophysical and economic models) with the aim of supporting policy-decision making. Although most of the IAMs are global, recent modeling developments have linked models across spatial scales.

Meta-analysis and systematic reviews: Meta-analysis involves a systematic review of published studies, and can include statistical analysis of results to obtain general conclusions. Some recent meta-analyses include Challinor et al. (2014), using 1,700 studies to assess the impacts of climate change on maize, wheat, and rice yields. Corbeels et al. (2014) and Rusinamhodzi et al. (2011) have conducted meta-analysis of conservation agriculture studies.

4.2 Modeling studies: scales, impact areas, and indicators

Table 2 presents examples of case studies using the approaches and models discussed in the previous sections and identifies the impact areas and types of indicators used. The case studies presented illustrate how they can inform priority setting at each scale. This is not a comprehensive review; rather, the goal is to illustrate studies across the types of methods described in the previous section and the scale they have been implemented. To focus the discussion, we consider how they map into the five impact areas identified by the CGIAR.

Table 2.

Selected case studies: scale, models, impact areas, and indicators.

			Areas of impact
Case study	Scale	Model(s)	Poverty	Nutrition	Gender	Environment	Climate	Indicators	Approach	CGIAR Center
Herrero et al. (2014)	Cross-scale	Logit, CLUE-S, IMPACT-Household, LP, DSSAT				X		Land use, yields	Integrated assessment modeling	CCAFS, ILRI
Schlenker et al. (2006)	Farming system	Hedonic, Ricardian						Farm value, yield	Econometrics	N
Guijt (1998)	Farming system	Qualitative methods, M&E				X		Soil erosion, income	Qualitative approaches	N
Sain et al. (2017)	Farming system	Cost–benefit analysis, qualitative methods, @RISK				X		Profits, CO2 sequestration, labor	Cost–benefit/surplus	CIAT, CCAFS
Lamanna et al. (2016)	Farming system	Risk-Household-Option (RHO)		X		X		Food security, risk of extreme events, crop productivity	Qualitative–quantitative approach	CCAFS
Giller et al. (2011)	Farming system	NUANCES, FARMSIM, HEAPSIM, FIELD				X		Soil fertility, crop yield	Simulation modeling	CIAT, ILRI, AFRICARICE
Notenbaert et al. (2017)	Farming system, global	GIS, GLEAM, farm-scale model				X	X	GHG emissions, soil quality, crop and dairy productivity, land use	Integrated assessment modeling	CIAT, ILRI
Rosegrant et al. (2014)	Global	IMPACT, DSSAT		X		X	X	Yields, N losses, water productivity, trade, risk of hunger, malnutrition, income	Integrated assessment modeling	IFPRI
Borgomeo et al. (2016)	Global	MOMP, CATCHMOD				X	X	Water resources, financial costs	Mathematical programming/optimization methods	N
Herrero et al. (1999)	Global	MCDM				X		Gross margin, dairy production	Mathematical programming/optimization methods	N
Hareau et al. (2014)	Global	Economic surplus	X			X		Poverty, production	Cost–benefit/surplus	CIP
Challinor et al. (2014)	Global	Meta-analysis					X	Crop yields, income, emissions	Meta-analysis/systematic review	CCAFS
Havlík et al. (2014)	Global	GLOBIOM, GFM, EPIC, CENTURY		X		X	X	GHG emissions, food security, calorie sources, feed, livestock and crop productivity	Integrated assessment modeling	ILRI, CIFOR, CCAFS, CIAT
Havlík et al. (2011)	Global	GLOBIOM, EPIC				X	X	GHG emissions, land use, energy	Integrated assessment modeling	N
Weindl et al. (2015)	Global	LPJmL, MAgPIE				X	X	Crop yields, rangeland use	Integrated assessment modeling	N
Rosegrant et al. (2017)	Global	IMPACT		X		X	X	Hunger, calories, food security, crop yields, GHGs	Integrated assessment modeling	IFPRI
Kristjanson et al. (1999)	Multi-country	GIS- Economic surplus				X		Income	Cost–benefit/surplus	ILRI
Nedumaran et al. (2014)	Multi-country	Economic surplus				X		Yields	Cost–benefit/surplus	ICRISAT
Twyman (2018)	Multi-country	Qualitative methods			X			Women's participation in farm activities and decision making	Qualitative approaches	CIAT
Claessens et al. (2012)	Farming system and national	TOA-MD	X				X	Crop and dairy yields, income	Simulation modeling (parsimonious)	CIP
Shirsath et al. (2017)	Farming system and national	InfoCrop, cost–benefit analysis				X		Crop yields, income, emissions	Cost–benefit/surplus	CCAFS
Valdivia et al. (2017)	Farming system and national	TOA-ME, DSSAT	X			X	X	Yields, income	Simulation modeling (spatially explicit)	N
Shikuku et al. (2017)	Farming system and national	TOA-MD	X	X		X	X	Income, poverty, adoption rate, food security, GHG emissions, yields	Simulation modeling (parsimonious)	CIAT
Antle et al. (2015b)	Farming system and national	TOA-MD	X	X		X		Income, food and protein consumption, yields	Simulation modeling (parsimonious)	WorldFish
Groot et al. (2012)	Farming system and national	FARMDesign				X		Profits, yields, soil N losses	Mathematical programming/optimization methods	N
Wossen and Berger (2015)	Farming system and national	MPMAS	X	X		X	X	Yield, income, poverty, food consumption	Mathematical programming/optimization methods	CIAT
Holzkämper et al. (2015)	Farming system and national	CROPSYST, MOMP				X	X	Yields, leaching	Mathematical programming/optimization methods	N
Van den Bergh (2004)	Farming system and national	Cost–benefit analysis				X	X	Yields, GHG emissions	Cost–benefit/surplus	N
Leary (1999)	Farming system and national	Cost–benefit analysis					X	N/A	Cost–benefit/surplus	N
Kumar et al. (2018)	Farming system and national	Cost–benefit analysis					X	Income	Cost–benefit/surplus	CCAFS-ICRISAT
Shiferaw et al. (2008)	Farming system and national	DREAM		X		X		Income,	Cost–benefit/surplus	ICRISAT, IFPRI
Wander et al. (2004)	Farming system and national	Economic surplus				X		NPV, BCR	Cost–benefit/surplus	N
Mendelsohn et al. (1994)	Farming system and national	Ricardian, DSSAT				X	X	Land value	Econometrics
Antle and Capalbo (2001)	Farming system and national	Econometric process model, crop models						Net returns, prices, and production (supply curves)	Integrated assessment modeling	N
Khatri-Chhetri et al. (2017)	Farming system and national	Multinomial Probit			X	X		Income, gender	Econometrics	CCAFS, IFPRI, CIMMYT
Mwongera et al. (2017)	Farming system and national	CSA-rapid appraisal			X			Men/women participation, income, Well-being index, assets index	Qualitative approaches	CIAT, IITA
AgMIP: implemented in several countries in SSA and SA, in collaboration with ICRISAT, CIAT, CIMMYT, and ICRAF (https://agmip.org)	Farming system and national	APSIM, DSSAT, TOA-MD	X			X	X	Income, poverty, adoption rates, vulnerability, gains and losses	Integrated assessment modeling	ICRISAT, CIAT, CIMMYT

Field level: Biophysical process-based models that simulate crop or livestock yields are widely used. Among the most important ones are the multiple crop simulation models embedded in the Decision Support System for Agrotechnology Transfer (DSSAT; Hoogenboom et al. 2019), the Agricultural Production Systems sIMulator (APSIM; Keating et al. 2003), Cropping Systems Simulation Model (CROPSYST; Stöckle, Donatelli, and Nelson 2003), Environmental Policy Integrated Climate (EPIC; Williams and Singh 1995), RUMINANT (Herrero et al. 2013), LIVESIM (Rufino et al. 2009), and LIFE-SIM (León Velarde et al. 2006).

Farm level: Economic models are used to estimate farmers’ livelihoods as well as farm profitability. Often these models are linked to biophysical models to transfer field-level data (e.g. crop yields) and to environmental models (e.g. pesticide leaching). van Wijk et al. (2014) reviewed household- and farm-level models and evaluated and compared their attributes and approaches. At the landscape scale, biophysical models can simulate processes over large areas (e.g. watershed). The Soil Water Assessment Tool (Arnold et al. 1998), for example, simulates impacts of land use on water quantity and quality and sedimentation. The Integrated Valuation of Ecosystem Services and Trade-offs Tool (Tallis and Polasky 2009) estimates carbon sequestration and other ecosystem services in mixed-use landscapes using a suite of different models.

Some economic models at this scale can be used to simulate potential adoption of agricultural technology or policy interventions. The Trade-off Analysis for Multi-dimensional Impact Assessment Model (TOA-MD; Antle and Valdivia 2020a; Antle, Stoorvogel, and Valdivia 2014) can simulate adoption rates of alternative technologies in heterogeneous populations of farms and the associated social, economic, and environmental impacts.

National scale: Partial equilibrium economic models of the agricultural sector, and computable general equilibrium (CGE) models of the entire economy, are being used at the national level. For example, the IMPACT model (Rosegrant et al. 2008) has been adapted for national-level analysis of policy interventions for Senegal and South Africa, and the Global Biosphere Management (GLOBIOM) model developed by the International Institute of Applied Systems Analysis (IIASA; Havlík et al. 2011) has been adapted for national analysis for Brazil and Ethiopia. Another national-level model that has been applied to multiple countries in the context of climate change and adaptation assessments is the FAO Modelling System for Agricultural Impacts of Climate Change to Support Decision-Making in Adaptation (Kuik et al. 2011). The FABLE consortium has developed a generic national model that is being used by national modeling teams to assess sustainable development pathways with climate change (https://www.foodandlandusecoalition.org/fable/).

Global level: More than ten major economic modeling systems have been used for global analysis of prices, trade, and policy, and are also used for climate impact assessment (Nelson et al. 2014). CGE models that capture interactions between agriculture and other sectors that affect supply, demand, and crop price formation as well as partial equilibrium models that calculate direct and indirect effects of agricultural productivity change under different economic, climatic, and demographic scenarios have been widely used by CGIAR researchers. The IMPACT model (Rosegrant et al. 2008) developed by IFPRI has been used to assess agricultural policy impacts and to conduct long-term assessments of climate change on food, agriculture, and natural resources at global and regional scales. Similarly, the GLOBIOM model (Havlík et al. 2011) developed by the IIASA is frequently used to simulate competition for land use between agriculture, forestry, and bioenergy at the global level. Other global economic models include the CGE model developed by the Global Trade Analysis Project (Hertel, McDougall, and Itakura 2001), the Model of Agricultural Production and Its Impact on Environment (MAgPIE; Dietrich et al. 2019), and the Environmental Impact and Sustainability Applied General Equilibrium model (van der Mensbrugghe 2008).

Although there has been much progress developing and linking existing disciplinary models to assess impacts on various biological, physical, and economic outcomes (e.g. DSSAT and the soil carbon model CENTURY have been linked to integrate soil carbon and nitrogen processes to simulate crop yields and soil carbon dynamics), large gaps remain between models at various scales, e.g. farm system and landscape scales, and national scale for economic analysis of commodity and related markets, that prevent truly integrated assessments across scales. For example, most farm system models assume farms are price takers with no formal linkage to a market equilibrium model. A few exceptions include Laborte, Van Ittersum, and Van den Berg (2007), van Ruijven, O'Neill, and Chateau (2015), and Valdivia, Antle, and Stoorvogel (2012); however, further research is needed to address key aggregation and disaggregation issues (Antle, Stoorvogel, and Valdivia 2014).

4.3 Case studies from farm to global scales

This section presents two case studies that illustrate the types of analyses that are currently being done at various scales. These studies reflect the fact that stakeholder objectives vary according to the spatial and temporal scales, from local to national to global, and from growing season to the decadal progression of climate change and other global drivers. These scales in turn determine relevant indicators and the trade-offs that need to be evaluated. These case studies illustrate how FA and TOA can be used to guide project-level research design and data collection to support ongoing ME and periodic re-evaluation of research priorities (e.g. updating theory of change).

Economic, environmental, and social impact assessment of the East Africa Dairy Development Project in Kenya (Antle and Valdivia 2011): The goal of this study was to conduct an impact assessment of the practices promoted by the East Africa Dairy Development (EADD) Project, using baseline data collected by the International Livestock Research Institute (ILRI). This analysis was designed as a proof of concept for evaluation of research investments by the Bill and Melinda Gates Foundation, using an environmental accounting matrix with the TOA-MD. The analysis highlighted some of the complex economic, environmental, and social trade-offs and synergies that are likely to be associated with dissemination of the EADD practices at farm, subregional, and regional scales.

At the farm level, the EADD study shows that while farm income increases and poverty decreases, and total assets owned by both male and women increase, the share of assets controlled by women declines, and water use and GHG emissions increase. These results indicate that there are likely to be trade-offs between the economic benefits of livestock intensification and key social and environmental impacts that could be addressed in the research design for improvements in livestock management to reduce water use and GHG emissions. Inclusive technology dissemination could be designed to mitigate possible adverse gender effects of the new technology.

The analysis shows that different EADD implementation scenarios result in different potential adoption rates and important trade-offs and synergies across the outcome indicators. Increasing income reduces poverty and increases infant milk consumption as well as efficiency in the use of water and methane emissions. Total water use and methane emissions also increase. The EADD study shows that while farm income increases, the poverty rate decreases and total assets owned by both male and women increase, but the share of assets controlled by women is predicted to decline. This kind of information can support research priority setting and resource allocation decisions. For example, these results indicate that development and investment strategies to mitigate possible adverse gender effects of the new technology are needed (see Fig. 5).

Figure 5.

Trade-offs between farm income, infant milk consumption, women's asset share, poverty rate, CH₄ emissions, CH₄ efficiency, water use, and water efficiency. Elaborated with data from Antle and Valdivia (2011).

The AgMIP: the Crop–Livestock Intensification Project—ICRISAT (Homann-Kee Tui et al. 2020): This study applied a new protocol-based approach to climate impact and adaptation analysis to generate actionable information for policy decision making, developed by the AgMIP and collaborators (AgMIP 2017). Using the multiscale, multi-modeling approach described in Fig. 3, a climate change and adaptation impact assessment for smallholder farming systems in Zimbabwe drylands was carried out under current and future climate and socioeconomic conditions. The study produced indicators of vulnerability, defined as the proportion of farm households that may lose income due to climate change; economic gains or losses as well as the net economic impact; changes in mean farm returns; changes in the poverty rate among farm households; changes in crop and livestock yields; and potential adoption rates of proposed innovations and adaptation packages co-developed with local stakeholders. The foresight component of this study developed plausible future socioeconomic conditions, called Representative Agricultural Pathways (RAPs; Valdivia et al. 2015, 2020).

Crop simulations highlighted the importance of soils in quantifying the impacts of climate change on crop yields. The results show that soils with higher organic carbon and water-holding capacity will buffer the impacts better under climate change than soils with low productivity. Groundnuts and fodder legumes were identified as important commodities to offset climate change impacts for the extremely poor households. Livestock simulations showed that farms with large herds were the most impacted by climate change. Improving the feed base through feed production and marketing, and the use of stock feed, is critical. One key implication of this analysis is that even in vulnerable regions like the drylands in Zimbabwe, there are farm households that may lose and others that may gain from climate change, and that socioeconomic conditions play an important role in the assessment of vulnerability. Figure 6 shows that there are gainers and losers from climate change. The mean net economic impact under future conditions improves (3.18 per cent) compared to current conditions where, on average, losses are greater than gains. However, even with a positive net economic impact in the aggregate, the proportion of farm households vulnerable to economic losses is still large (e.g. 53 and 42 per cent under current and future conditions, respectively, in the region). Thus, reporting only net economic impacts (as in BCA) obscures the distribution of impacts and vulnerabilities.

Figure 6.

Proportion of vulnerable farm households and mean net economic impact for Nkayi, Zimbabwe. Elaborated with data from Homann-Kee Tui et al. (2020).

The AgMIP case study in Zimbabwe (Homann-Kee Tui et al. 2020) shows how TOA and FA can be used to inform multilevel stakeholders (from farming system to national level) about the likely consequences of following different development pathways. It also demonstrates that the process of co-designing pathways and adaptation strategies with stakeholders increased the demand for science-based information to support decision making. The TOA also showed that diversification of food, cash, and feed crops, like legumes and small grains, can improve income and nutrition and empower youth and women in the fragile drylands. The CGIAR Research Program on Grain Legumes and Dryland Cereals Agri-food Systems, managed by ICRISAT, can play a key role in the transformation of the crop–livestock system of this region. The AgMIP multi-model approach helped stakeholders to explore the potential for new technologies and farming strategies to enable farm households to adapt to climate change. The result was a portfolio of options to support investments in climate-smart agriculture.

The case studies presented above show the potential opportunities for using TOA and FA as tools for priority setting and project evaluation. One key implication is that implementation of FA and TOA into project design—as embodied in Fig. 1—has the potential of providing integrated assessments of economic, environmental, and social impacts at low cost and in a timely manner. This helps stakeholders to evaluate investment options and implications of paradigm shifts, comprehend different user perspectives and preferences for equitable solutions, and set priorities and allocate resources toward systems transformation. Other detailed cases can be found in Antle and Valdivia (2020b).

4.4 Representation of sustainable development indicators in TOA modeling

The preceding sections and Table 2 show the diverse use of tools at different scales and the features of the various models used. However, most of the studies focus on indicators that correspond to one or two impact areas. The EADD study is one of the few that incorporates indicators in all three dimensions. One explanation for this gap in most studies is the disciplinary orientation of the researchers carrying out projects. For example, projects led by agronomists or economists may lack the disciplinary knowledge needed to recognize the importance of impacts in environmental or social dimensions, as well as the expertise to identify appropriate indicators, design efficient data collection for those indicators, or implement appropriate models for simulation.

In this section, we discuss the capabilities of current modeling approaches to represent key impact areas of all three dimensions of sustainable development, and identify needed improvements.

Nutrition and food security: Most current models use simple indicators such as per-capita food consumption to represent nutrition and food security, but do not incorporate the indicators that have been developed to characterize the key determinants of food security (availability, access, stability, and utilization) and nutritional aspects such as diet diversity. A critical limitation is the lack of household food consumption data; thus, there is the need to develop methods to collect these data with new approaches that can capture the dynamics of household consumption (e.g. mobile apps). Most research now utilizes multiple indicators to measure nutritional diversity, which is becoming important to assess sustainable diets beyond assessments based on kilocalories (Müller, Elliott, and Levermann 2014). Recent emerging research on sustainable healthy eating behavior seeks to integrate eating or consumption attitudes and behavior toward changes in diets (Fanzo et al. 2012; Geiger, Fischer, and Schrader 2018). Additional research is needed to bring these dimensions into quantitative models.

Poverty and income distribution: Within agricultural populations, estimates of some components of farm household income are usually included in most economic household or farm models. However, a complete characterization of farm income sources (e.g. off-farm income) is needed to accurately assess poverty and income distribution in a population of farms. In addition, the distribution of income among household is required to estimate indicators such as poverty rates. The headcount poverty rate (i.e. the proportion of households below the poverty line) is the most frequent indicator used in TOA. Other indicators such as the poverty gap (i.e. the degree to which individuals are below the poverty line) and food-based poverty indicators (Smith and Urey 2002; Antle, Adhikari, and Price 2015) can be used to represent other dimensions of poverty. Going beyond farm households to other rural and urban populations involves broader data and models that are typically highly aggregated and thus not capable of representing income distributions or predicting changes in poverty rates. Linking disaggregate data and models with more aggregate models is the solution to this challenge that is currently at the frontier of research.

Gender and age: Despite the growing research on gender impacts and related outcomes, there is little representation of gender in most models, at least quantitatively. Similarly, analysis of children's labor and their nutrition and health has also been limited, due to lack of data. Distributional impacts of new technologies or policy interventions on gender and intra-household equity, including asset ownership, health, education, and nutrition, could be incorporated in quantitative or qualitative impact assessment models with better age- and sex-disaggregated data that are context specific.

Climate: Climate (and weather) has been included in many biophysical and economic models to assess the impacts of climate change on crop yields. However, many of these analyses are crop based. In order to have a better assessment of the vulnerability of farms to climate change, farm and household system-based analyses are needed, as illustrated by Case Study 2 in Section 4.3.

Environment: Field- or point-level agricultural models (e.g. crop simulation models) can capture the water, soil carbon, nitrogen, and nutrients, and environment fluxes. At this level, these models can be linked to economic models to assess the trade-offs between socioeconomic and environmental outcomes. For example, Valdivia, Antle, and Stoorvogel (2012) link the TOA–ME economic model to the DSSAT crop simulation model and the nutrient monitoring model (De Jager, Nandwa, and Okoth 1998) to assess the trade-offs between maize production, poverty, and soil nutrient depletion. At the landscape scale, upscaling the field-level processes increases the complexity of the system. Process-based models can be used to simulate hydrology, sediment and contaminant transport and cycling (pesticides, bacteria, and nutrients) in soils and streams, and crop or vegetative uptake, growth, and yields.

4.5 Improving models and enabling their use

In addition to better representation of all impact areas, there are many issues that arise in the use of models for technology impact assessments and TOA. Here, we summarize some recent research that has identified areas for data and model improvements (Antle, Jones, and Rosenzweig 2017; Jones et al. 2017).

Linking models across disciplines (e.g. crop–livestock–economic–environmental models): In order to link different models at farm system level, protocols and tools need to be created for standardized inputs and outputs (i.e. data and model harmonization).

Linking models across scales: Methods are required to link household or farm system-level economic models to market equilibrium models (e.g. partial or general equilibrium models). There is need to address aggregation and disaggregation issues.

Behavioral assumptions: Most of the economic models are based on profit-maximization assumptions (in some cases, adjusted for risk). However, recent literature on behavioral economics and risk modeling could be incorporated in the TOA.

Economic model intercomparison and improvement: The wide array of household- and farm-level economic models use different approaches and assumptions; however, they can be used for the same objective (e.g. assessing impacts of climate change on a specific farming system). Responses to each model are likely to be different. Thus, there is the need to understand the differences and explore options to improve models and be able to use model ensembles to capture the inherent uncertainty of the models. Intercomparison and improvement of crop simulation models have been conducted by the AgMIP. Also, a comparison activity that involved ten global economic models was carried out under AgMIP (Nelson et al. 2014).

Representation of pests and diseases: Crop and livestock process models are limited in the way they represent pest and diseases. However, there are advances on methods to incorporate pest and diseases in modeling approaches (statistical and process-based approaches).

Documentation and availability: Models and tools should be publicly available and well documented. This is crucial for capacity building and knowledge transfer. Likewise, model or method improved should also be documented.

FA and TOA: A stronger integration of science and stakeholder-based knowledge is needed to enable priority setting and to support decision-making processes effectively, as illustrated in Fig. 1 (Valdivia et al. 2020). At global level, a range of socioeconomic trends have been established by scientists and stakeholders (Shared Socioeconomic Pathways) and are now being widely used to represent the range of plausible future trajectories of population growth, GDP, and other global drivers. The CGIAR Research Program on Climate Change, Agriculture and Food Security (CCAFS) has developed similar multi-country future scenarios for West and East Africa (Palazzo et al. 2017; Vervoort, Antle, and Stoorvogel 2014). There is a need to develop more detailed national pathways specific to individual countries’ circumstances. Similarly, for analysis at the farming system level, more detailed pathways are needed to provide input parameters to crop, livestock, and economic models. For climate impact and adaptation analysis, AgMIP has established participatory methods to develop RAPs (Valdivia et al. 2015, 2020). Similar methods can be used for any type of forward-looking technology impact assessment, and can also be used to develop national-level pathways.

Data: A critical data limitation is that most available data are collected for disciplinary research and are often not readily accessible, or not suitable for use in impact assessment due to differences in spatial and temporal scales or lack of adequate documentation (Antle, Jones, and Rosenzweig 2017). For example, multiple detailed datasets exist on crop production or yields, but since the goal is to assess changes in crop yields or capture other biophysical processes, key economic data (e.g. production costs) are frequently not recorded. A relatively new initiative within the CGIAR is the Rural Household Multi-Indicator Survey (RHoMIS; van Wijk et al. 2020). RHoMIS is a tool that aims at reducing the cost and time required to carry out and analyze surveys. It provides a structure and standard format to collect data that describe farm productivity and practices, nutrition, food security, gender equity, climate, and poverty. However, one limitation is that it does not include economic data such as production costs that are critical for economic analysis and TOA. There is the need to advance data standards with the aid of digital technologies for collection. The Platform for Big Data in Agriculture of the CGIAR has defined a common core of cross-sectional household survey as a first building block for a standardization of survey data (van Wijk et al. 2019). The CGIAR's Big Data initiative is working toward the goal for agricultural data to meet desirable standards (findable, accessible, interoperable, ethical, and reproducible) through several actions: (1) identification of key concepts, indicators, and questions commonly addressed in socioeconomic surveys; (2) developing a socioeconomic ontology, building on existing ontologies and focusing on the key concepts, indicators, and questions from the socioeconomic team; (3) developing a standard for documenting and archiving metadata; and (4) identifying best practices in electronic data capture and development of data capture tools.

Capacity building: A strategy for capacity building is needed to enable the priority setting process that incorporates FA and TOA in agricultural research organizations (Fig. 1). The objective of the capacity building should include a research organization's management and its agricultural research scientists and stakeholders. There is a need to build capacity within research organizations for in-house modeling tools as well as for modeling tools developed by partner institutions. Examples are established modeling groups like DSSAT (DSSAT Foundation); TOA-MD Impact Assessment Courses offered by Oregon State University; IFPRI's IMPACT modeling group; and the Global Trade Analysis Project operated by Purdue University.

Private and public partnerships: Rapidly advancing genetic and digital technologies developed in the private sector mean that public sector agricultural research organizations increasingly need to establish strategic partnerships with the private sector. These partnerships could improve access and knowledge to products that are not public but that can contribute to the process of priority setting and development of innovations as public goods. For example, digital innovation in the private sector can support the creation, use, combination, analysis, and sharing of agricultural and other data in digital format to improve the sustainability and productivity of agriculture and food systems (OECD 2019).

5 FA and TOA: opportunities and challenges

We conclude that the currently available FA (Lentz 2020; Zurek, Hebinck, and Selomane 2020) and the TOA methods and tools discussed in this paper present an unprecedented opportunity for national and international public good research organizations aiming to set priorities among potential innovations to balance inevitable trade-offs and exploit synergies along sustainable development pathways. FA and TOA can help accomplish efficient management through coordination of research design and evaluation at the project level with identified global, regional, and national priorities. Achieving these results also will require substantial improvements in data and methods.

Based on recent experience with FA and TOA, a number of significant challenges will need to be addressed as they are integrated into a research organization's priority setting and research evaluation (Fig. 1). A key element in TOA is collaboration between scientists and stakeholders to identify sets of minimally sufficient impact indicators at global (or national) and project scales. Reaching a consensus on a relatively small number will be a major but critical challenge. A key finding of the Lentz (2020) and Zurek, Hebinck, and Selomane (2020) studies is that global-scale foresight studies do not adequately represent impacts in several dimensions of sustainable development—nutrition and food security; poverty reduction, livelihoods, and jobs; and gender dimensions, youth, and social inclusion. This is also true for many important environmental impacts, and is a feature (or limitation) of the global and national impact assessment models discussed in this paper. This condition is due in part due to the lack of consensus on impact areas and indicators, but mainly due to a lack of analysis at disaggregate scales (farm, eco-regional) that can be meaningfully aggregated to national, regional, and global scales. Project-level implementation of FA and TOA will help fill this gap in available data at higher levels of aggregation.

A major challenge in the implementation of FA and TOA of complex, multiscale systems is to produce information that is low cost and timely. Based on experience applying FA and TOA, analysts need to embrace the principle of parsimony: information to support decisions under uncertainty needs to be minimally sufficient to identify options that are likely to meet goals and balance trade-offs across multiple impact dimensions. Accordingly, the need for timely information to support decisions argues for parsimonious quantitative modeling and analysis. Priority setting processes should be designed to be cost-effective and fit-to-purpose, namely, to provide an objective evidence base to guide and justify priority setting decisions. As illustrated by the Rockefeller pesticide studies, the careful selection of impact indicators is a key element to addressing complexity through sufficiency and parsimony: if acute health impacts of toxic pesticides on farmer health show that farmers are exposed to unacceptably high neurological risks, evidence of chronic health impacts such as cancer risks is not necessary to justify research that can help farmers to reduce the use of toxic pesticides while achieving other goals such as poverty reduction and food security.

Acknowledgments

This article was adapted from a commissioned report that was funded by the CGIAR Independent Science for Development Council (ISDC). The authors wish to thank the members of the ISDC for their comments and suggestions during the development of that report.

Appendix

Table A1.

Economic indicators and SDGs.

	Economic indicators	Unit	SDG
Farm	Crop or livestock productivity (yield)	Quantity per hectare or quantity per animal	SDG1: No Poverty; SDG2: Zero Hunger; SDG8: Promote Sustained Inclusive and Sustainable Economic Growth; SDG13: Climate Action
	Financial condition	Debts/assets (per cent)
	Farm income	Currency units per farm, per hectare, per animal unit
	Technology: improved genetics; purchased inputs (seeds, fertilizers, pesticides); mechanical power and implements; information technology	Use/non-use; application rates (quantity per hectare)
	Diversification or resilience	Crop or livestock species diversity index, drought or disease tolerance
Household	Money income	All farm and non-farm money income (currency units/time)
	Full income	Net value of all farm and household production and labor plus non-farm money income (currency units/time)
	Poverty	Percentage of individuals or households with consumption or income below poverty line
	Vulnerability	Possibility of suffering a decline in well-being due to an adverse shock

Source: Based on Antle and Ray (2020).

Table A2.

Environmental indicators and SDGs.

	Indicator	Unit	SDG
Soil	Soil organic matter	Percentage of soil	SDG2: Zero Hunger; SDG6: Ensure Availability and Sustainable Management of Water and Sanitation; SDG12: Responsible Consumption and Production; SDG15: Halt Biodiversity Loss
	Soil fertility	pH, macro–micro nutrient balance
	Soil erosion	kg/ha
Water	Depth of groundwater	meters
	Water quality	pH, salinity
	Dissolved oxygen in water	per cent, mg pollutant/liter, ppm
	Heavy metal concentration in water	per cent, mg pollutant/liter, ppm
Emissions	Carbon dioxide (CO2), methane (CH4), nitrous oxide (N2O)	kg GHG/year
	Air pollution	Air quality indices
Land use	Increase in forest cover	Share of land converted to protected forests, share of degraded land recovered
	Conservation of fragile ecosystems	Conserved ecosystems in coastal, mountains, and island systems
Biodiversity	Species richness	Gamma diversity: count of species in a region
	Protection for terrestrial and freshwater biodiversity	Share of land protected

Table A3.

Social indicators for human development, equity, and justice and SDGs.

	Indicator	Unit	SDG
Food security	Availability of culturally relevant food	Calories/capita, expenditure/capita	SDG2: Zero Hunger
	Access to food	Household Food Insecurity Access Scale, Food Insecurity Access Scale
	Diet diversity	Food Consumption Score, Household Diet Diversity Scale
	Food safety	Percentage of population with access to clean drinking water, number of food contamination cases due to pathogens and agri-chemicals; cases of foodborne illness or death
Nutrition	Undernutrition	Percentage of children stunted, wasted, or underweight, percentage of those with micronutrient deficiency
	Overnutrition	Percentage of obese, percentage of calories from saturated fats
Health	Maternal and child mortality	Maternal mortality ratio, infant mortality rate	SDG3: Good Health & Well-being
	Mental health	Cases of illness or deaths
Women's empowerment	Women's land ownership, control over assets, and autonomy in the household	Percentage of land owned by women, percentage of assets controlled by women	SDG5: Gender Equality
	Pay gap	Relative difference in wages of men and women
	Intra-household resource allocation to female members	Difference in food allocation, and educational investments between girls and boys
	Women's Empowerment in Agriculture Index	Index
Safe working conditions	Worker safety	Number of injuries or deaths	SDG8: Decent Work and Economic Growth
	Exposure to harmful chemicals	Percentage of workers without protective gear, percentage of workers exposed to harmful chemicals
Community	Viability	Age distribution	SDG11: Sustainable Cities and Communities
		Educational, medical, and social services
Animal welfare	Confinement practices	Percentage of those confined
	Animal health	Disease and mortality rates