Research impact assessment in agriculture — A review of approaches and impact areas

Research has a role to play in society’s endeavour for sustainable development. This is particularly true for agricultural research, since agriculture is at the nexus between numerous sustainable development goals. Yet, generally accepted methods for linking research outcomes to sustainability impacts are missing. We conducted a review of scientific literature to analyse how impacts of agricultural research were assessed and what types of impacts were covered. A total of 171 papers published between 2008 and 2016 were reviewed. Our analytical framework covered three categories: (1) the assessment level of research (policy, programme, organization, project, technology, or other); (2) the type of assessment method (conceptual, qualitative, or quantitative); and (3) the impact areas (economic, social, environmental, or sustainability). The analysis revealed that most papers (56%) addressed economic impacts, such as cost-effectiveness of research funding or macroeconomic effects. In total, 42% analysed social impacts, like food security or aspects of equity. Very few papers (2%) examined environmental impacts, such as climate effects or ecosystem change. Only one paper considered all three sustainability dimensions. We found a majority of papers assessing research impacts at the level of technologies, particularly for economic impacts. There was a tendency of preferring quantitative methods for economic impacts, and qualitative methods for social impacts. The most striking finding was the ‘blind eye’ towards environmental and sustainability implications in research impact assessments. Efforts have to be made to close this gap and to develop integrated research assessment approaches, such as those available for policy impact assessments.


Introduction
Research has multiple impacts on society. In the light of the international discourse on grand societal challenges and sustainable development, the debate is reinforced about the role of research on economic growth, societal well-being, and environmental integrity (1). Research impact assessment (RIA) is a key instrument to exploring this role (2).
A number of countries have begun using RIA to base decisions for allocation of funding on it, and to justify the value of investments in research to taxpayers (3). The so-called scientometric assessments with a focus on bibliometric and exploitable results such as patents are the main basis for current RIA practices (4)(5)(6). However, neither academic values of science, based on the assumption of 'knowledge as progress', nor market values frameworks ('profit as progress') seem adequate for achieving and assessing broader public values (7). Those approaches do not explicitly acknowledge the contribution of research to solving societal challenges, although they are sufficient to measure scientific excellence (8) or academic impact.
RIA may however represent a vital element for designing socially responsible research processes with orientation towards responsibility for a sustainable development (9,10). In the past, RIAs occurred to focus on output indicators and on links between science and productivity while hardly exploring the wider societal impacts of science (11). RIA should entail the consideration of intended and nonintended, positive and negative, and long-and short-term impacts of research (12). Indeed, there has been a broadening of impact assessments to include, for example, cultural and social returns to society (13). RIA is conceptually and methodologically not yet sufficiently equipped to capture wider societal implications, though (14). This is due to the specific challenges associated with RIA, including inter alia unknown time lags between research processes and their impacts (15)(16)(17). Independent from their orientation, RIAs are likely to influence research policies for years to come (18).
Research on RIA and its potential to cover wider societal impacts has examined assessment methods and approaches in specific fields of research, and in specific research organizations. The European Science Foundation (19) and Guthrie et al. (20) provided overviews of a range of methods usable in assessment exercises. They discuss generic methods (e.g. economic analyses, surveys, and case studies) with view to their selection for RIAs. Methods need to fit the objectives of the assessment and the characteristics of the disciplines examined. Econometric methods consider the rate of return over investment (21), indicators for 'productive interactions' between the stakeholders try to capture the social impact of research (22), and case study-based approaches map the 'public values' of research programmes (8,23). No approach is generally favourable over another, while challenges exist in understanding which impact areas are relevant in what contexts. Penfield et al. (6) looked at the different methods and frameworks employed in assessment approaches worldwide, with a focus on the UK Research Excellence Framework. They argue that there is a need for RIA approaches based on types of impact rather than research discipline. They point to the need for tools and systems to assist in RIAs and highlight different types of information needed along the output-outcome-impact-chain to provide for a comprehensive assessment. In the field of public health research, a minority of RIAs exhibit a wider scope on impacts, and these studies highlight the relevance of case studies (24). However, case studies often rely on principal investigator interviews and/or peer review, not taking into account the views of end users. Evaluation practices in environment-related research organizations tend to focus on research uptake and management processes, but partially show a broader scope and longer-term outcomes. Establishing attribution of environmental research to different types of impacts was identified to be a key challenge (25). Other authors tested impact frameworks or impact patterns in disciplinary public research organizations. For example, Gaunand et al. (26) analysed an internal database of the French Agricultural research organization INRA with 1,048 entries to identify seven impact areas, with five going beyond traditional types of impacts (e.g. conservation of natural resources or scientific advice). Besides, for the case of agricultural research, no systematic review of RIA methods exists in the academic literature that would allow for an overview of available approaches covering different impact areas of research.
Against this background, the objective of this study was to review in how far RIAs of agricultural research capture wider societal implications. We understand agricultural research as being a prime example for the consideration of wider research impacts. This is because agriculture is a sector which has direct and severe implications for a range of the UN Sustainable Development Goals. It has a strong practice orientation and is just beginning to develop a common understanding of innovation processes (27).

Analytical framework
The analysis of the identified literature on agricultural RIA (for details, see next section 'Literature search') built on a framework from a preliminary study presented at the ImpAR Conference 2015 (28). It was based on three categories to explore the impact areas that were addressed and the design of RIA. In particular, the analytical framework consisted of: (1) the assessment level of research; (2) the type of assessment method; and (3) the impact areas covered. On the side, we additionally explored the time dimension of RIA, i.e. whether the assessment was done ex ante or ex post (see Fig. 1).
Agricultural research and the ramifications following from that refer to different levels of assessment (or levels of evaluation, (29)). We defined six assessment levels that can be the subject of a RIA: policy, programme, organization, project, technology, and other. The assessment level of the RIA is a relevant category, since it shapes the approach to the RIA (e.g. the impact chain of a research project differs to that at policy level). The assessment level was clearly stated in all of the analysed papers and in no case more than one assessment level was addressed. Articles were assigned to the policy level, if a certain public technology policy (30) or science policy, implemented by governments to directly or indirectly affect the conduct of science, was considered. Exemplary topics are research funding, transfer of research results to application, or contribution to economic development. Research programmes were understood as instruments that are adopted by government departments, or other organizational entities to implement research policies and fund research activities in a specific research field (e.g. programmes to promote research on a certain crop or cultivation technique). Articles dealing with the organizational level assess the impact of research activities of a specific research organization. The term research organization comprises public or private research institutes, associations, networks, or partnerships (e.g. the Consultative Group on International Agricultural Research (CGIAR) and its research centres). A research project is the level at which research is actually carried out, e.g. as part of a research programme. The assessment of a research project would consider the impacts of the whole project, from planning through implementation to evaluation instead of focusing on a specific project output, like a certain agricultural innovation. The technology level was considered to be complementary to the other assessment levels of research and comprises studies with a strong focus on specific agricultural machinery or other agricultural innovation such as new crops or crop rotations, fertilizer applications, pest control, or tillage practices, irrespective of the agricultural system (e.g. smallholder or high-technology farming, or organic, integrated, or conventional farming). The category 'other' included one article addressing RIA at the level of individual researchers (see (31)).
We categorized the impact areas along the three dimensions of sustainable development by drawing upon the European Commission's impact assessment guidelines (cf. (32)). The guidelines entail a list of 7 environmental impacts, such as natural resource use, climate change, or aspects of nature conservation; 12 social impacts, such as employment and working conditions, security, education, or aspects of equity; and 10 economic impacts, including business competitiveness, increased trade, and several macroeconomic aspects. The European Commission's impact assessment guidelines were used as a classification framework because it is one of the most advanced impact assessment frameworks established until to date (33). In addition, we opened a separate category for those articles exploring joint impacts on the three sustainability dimensions. Few articles addressed impacts in two sustainability dimensions which we assigned to the dominating impact area.
To categorize the type of RIA method, we distinguished between conceptual, qualitative, and quantitative. Conceptual analyses include the development of frameworks or concepts for measuring impacts of agricultural research (e.g. tracking of innovation pathways or the identification of barriers and supporting factors for impact generation). Qualitative and quantitative methods were identified by the use of qualitative data or quantitative data, respectively (cf. (34)(35)(36)). Qualitative data can be scaled nominally or ordinally. It is generated by interviews, questionnaires, surveys or choice experiments to gauge stakeholder attitudes to new technologies, their willingness to pay, and their preference for adoption measures. The generation of quantitative data involves a numeric measurement in a standardized way. Such data are on a metric scale and are often used for modelling. The used categorization is rather simple. We assigned approaches which employed mixed-method approaches according to their dominant method. We preferred this over more sophisticated typologies to achieve a high level of abstraction and because the focus of our analysis was on impact areas rather than methods. However, to show consistencies with existing typologies of impact assessment methods (19,37), we provide an overview of the categorization chosen and give examples of the most relevant types of methods.
To additionally explore the approach of the assessment (38), the dimensions ex ante and ex post were identified. The two approaches are complementary: whereas ex ante impact assessments are usually conducted for strategic and planning purposes to set priorities, ex post impact assessments serve as accountability validation and control against a baseline. The studies in our sample that employed an ex ante approach to RIA usually made this explicit, while in the majority of ex post impact assessments, this was indicated rather implicitly.

Literature search
This study was performed as a literature review based on Thomson Reuters Web of Science TM Core Collection, indexed in the Science Citation Index Expanded (SCI-Exp) and the Social Sciences Citation Index (SSCI). The motivation for restricting the analysis to articles from ISI-listed journals was to stay within the boundaries of internationally accepted scientific quality management and worldwide access. The advantages of a search based on Elsevier's Scopus V R (more journals and alternative publications, and more articles from social and health science covered) would not apply for this literature review, with regard to the drawbacks of an index system based on abstracts instead of citation indexes, which is not as transparent as the Core Collection regarding the database definable by the user. We selected the years of 2008 to mid-2016 for the analysis (numbers last updated on 2 June 2016). First, because most performance-based funding systems have been introduced since 2000, allowing sufficient time for the RIA approaches to evolve and literature to be published. Secondly, in 2008 two key publications on RIA of agricultural research triggered the topic: Kelley, et al. (38) published the lessons learned from the Standing Panel on Impact Assessment of CGIAR; Watts, et al. (39) summarized several central pitfalls of impact assessment concerning agricultural research. We took these publications as a starting point for the literature search. We searched in TOPIC and therefore, the terms had to appear in the title, abstract, author keywords, or keywords plus V R . The search query 1 filtered for agricultural research in relation to research impact. To cover similar expressions, we used science, 'R&D', and innovation interchangeably with research, and we searched for assessment, evaluation, criteria, benefit, adoption, or adaptation of research.
We combined the TOPIC search with a less strict search query 2 in TITLE using the same groups of terms, as these searches contained approximately two-thirds non-overlapping papers. Together they consisted of 315 papers. Of these, we reviewed 282 after excluding all document types other than articles and reviews (19 papers were not peer-reviewed journal articles) and all papers not written in English language (14 papers). After going through them, 171 proved to be topic-relevant and were included in the analysis.

Results
The application of the analytical framework allowed for the classification of the 171 reviewed articles regarding the assessment level of the research, the impact areas considered, and the type of method applied. These results are structured in a quantitative matrix (Table 1).
In the agricultural RIA, the core assessment level of the reviewed articles was technology (39%), while the other levels were almost equally represented (with the exception of 'other'). Generally, most papers (56%) addressed economic research impacts, closely followed by social research impacts (42%); however, only three papers (2%) addressed environmental research impacts and only 1 of 171 papers addressed all three dimensions of sustainable development. Assessments at the level of research policy slightly emphasized social impacts over economic impacts (18 papers, or 58%), whereas assessments at the level of technology clearly focused primarily on economic impacts (46 papers, or 68%).
The methods used for agricultural RIA showed no preference for one method type (see Table 1). Approximately 31% of the papers assessed research impacts quantitatively, whereas 37% used qualitative methods. Conceptual considerations on research impact were applied by 32% of the studies. A noticeable high number of qualitative studies were conducted to assess social impacts. At the evaluation level of research policy and research programmes, we found a focus on quantitative methods, if economic impacts were assessed.
A non-exhaustive list of methods applied in the analysed studies is provided in Table 2. The list is meant to illustrate what is behind the three categories of conceptual, qualitative, and quantitative RIAs used in our review (method Type I) referring to typologies (method Type II) from the European Science Foundation (19) and RAND Europe (37).
Additionally, 37 ex ante studies, compared to 134 ex post studies, revealed that the latter clearly dominated, but no robust relation to any other investigated characteristic was found. Of the three environmental impact studies, none assessed ex ante, while the one study exploring sustainability impacts did. The share of ex ante assessments regarding social impacts was very similar to those regarding economic impacts. Within the assessment levels of research (excluding 'others' with only one paper), no notable difference between the shares of ex ante assessments occurred as they ranged between 13 and 28%.

Discussion
The most relevant outcome of the review analysis was that only 3 of the 171 papers focus on the environmental impacts of agricultural research. This seems surprising because agriculture is dependent on an intact environment. However, this finding is supported by two recent reviews: one from Bennett, et al. (40) and one from Maredia and Raitzer (41). Both note that not only international agricultural research in general but also research on natural resource management shows a lack regarding large-scale assessments of environmental impacts. The CGIAR also recognized the necessity to deepen the understanding of the environmental impacts of its work because RIAs had largely ignored environmental benefits (42).
A few papers explicitly include environmental impacts of research in addition to their main focus. Raitzer and Maredia (43) address water depletion, greenhouse gas emissions, and landscape effects; however, their overall focus is on poverty reduction. Ajayi et al. (44) report the improvement of soil physical properties and soil biodiversity from introducing fertilizer trees but predominantly measure economic and social effects. Cavallo, et al. (45) investigate users' attitudes towards the environmental impact of agricultural tractors (considered as technological innovation) but do not measure the environmental impact. Briones, et al. (46) configure an environmental 'modification' of economic surplus analysis, but they do not prioritize environmental impacts.
Of course, the environmental impacts of agricultural practices were the topic of many studies in recent decades, such as Kyllmar,et al. (47), Skinner, et al. (48), Van der Werf and Petit (49), among many others. However, we found very little evidence for the impact of agricultural research on the environment. A study on environmental management systems that examined technology adoption rates though not the environmental impacts is exemplarily for this (50). One possible explanation is based on the observation made by Morris,et al. (51) and Watts, et al. (39). They see impact assessments tending to accentuate the success stories because studies are often commissioned strategically as to demonstrate a certain outcome. This would mean to avoid carving out negative environmental impacts that conflict with, when indicated, the positive economic or societal impacts of the assessed research activity. In analogy to   policy impact assessments, this points to the need of incentives to equally explore intended and unintended, expected and nonexpected impacts from scratch (52). From those tasked with an RIA, this again requires an open attitude in 'doing RIA' and towards the findings of their RIA. Another possible explanation was given by Bennett, et al. (40): a lack of skills in ecology or environmental economics to cope with the technically complex and data-intensive integration of environmental impacts. Although such a lack of skills or data could also apply to social and economic impacts, continuous monitoring of environmental data related to agricultural practices is particularly scarce. A third possible explanation is a conceptual oversight, as environmental impacts may be thought to be covered by the plenty of environmental impact assessments of agricultural activities itself.
The impression of a 'blind eye' on the environment in agricultural RIA may change when publications beyond Web of Science TM Core Collection are considered (53) or sources other than peerreviewed journal articles are analysed (e.g. reports; conference proceedings). See, for example, Kelley, et al. (38), Maredia and Pingali (54), or FAO (55). Additionally, scientific publications of the highest quality standard (indicated by reviews and articles being listed in the Web of Science TM Core Collection) seem to not yet reflect experiences and advancements from assessment applications on research and innovation policy that usually include the environmental impact (56).
Since their beginnings, RIAs have begun to move away from narrow exercises concerned with economic impacts (11) and expanded their scope to social impacts. However, we only found one sustainability approach in our review that would cover all three impact areas of agricultural research (see (57)). In contrast, progressive approaches to policy impact assessment largely attempt to cover the full range of environmental, social, and economic impacts of policy (33,58). RIAs may learn from them.
Additionally, the focus of agricultural research on technological innovation seems evident. Although the word innovation is sometimes still used for new technology (as in 'diffusion of innovations'), it is increasingly used for the process of technical and institutional change at the farm level and higher levels of impact. Technology production increasingly is embedded in innovation systems (59).
The review revealed a diversity of methods (see Table 2) applied in impact assessments of agricultural research. In the early phases of RIA, the methods drawn from agricultural economics were considered as good standard for an impact assessment of international agricultural research (39). However, quantitative methods most often address economic impacts. In addition, the reliability of assessments based on econometric models is often disputed because of strong relationships between modelling assumptions and respective results.
Regarding environmental (or sustainability) impacts of agricultural research, the portfolio of assessment methods could be extended by learning from RIAs in other impact areas. In our literature sample, only review, framework development (e.g. key barrier typologies, environmental costing, or payments for ecosystem services), life-cycle assessment, and semi-structured interviews were used for environmental impacts of agricultural research.
In total, 42 of the 171 analysed papers assessed the impact of participatory research. A co-management of public research acknowledges the influence of the surrounding ecological, social, and political system and allows different types of stakeholder knowledge to shape innovation (60). Schut,et al. (36) conceptualize an agricultural innovation support system, which considers multi-stakeholder dynamics next to multilevel interactions within the agricultural system and multiple dimensions of the agricultural problem. Another type of participation in RIAs is the involvement of stakeholders to the evaluation process. A comparatively low number of six papers considered participatory evaluation of research impact, of them three in combination with impact assessment of participatory research.
Approximately 22% of the articles in our sample on agricultural research reported that they conducted their assessments ex ante, but most studies were ex post assessments. Watts, et al. (39) considered ex ante impact assessment to be more instructive than ex post assessment because it can directly guide the design of research towards maximizing beneficial impacts. This is particularly true when an ex ante assessment is conducted as a comparative assessment comprising a set of alternative options (61).
Many authors of the studies analysed were not explicit about the time frames considered in their ex post studies. The potential latency of impacts from research points to the need for ex post (and ex ante) studies to account for and analyse longer time periods, either considering 'decades' (62,63) or a lag distribution covering up to 50 years, with a peak approximately in the middle of the impact period (64). This finding is in line with the perspective of impact assessments as an ongoing process throughout a project's life cycle and not as a one-off process at the end (51). Nevertheless, ex post assessments are an important component of a comprehensive evaluation package, which includes ex ante impact assessment, impact pathway analysis, programme peer reviews, performance monitoring and evaluation, and process evaluations, among others (38).
RIA is conceptually and methodologically not yet sufficiently equipped to capture wider societal implications, though (14). This is due to the specific challenges associated with RIA, including inter alia unknown time lags between research processes and their impacts ( [15][16][17]. Independent from their orientation, RIAs are likely to influence research policies for years to come (18).
However, in the cases in which a RIA is carried out, an increase in the positive impacts (or avoidance of negative impacts) of agricultural research does not follow automatically. Lilja and Dixon (65) state the following methodological reasons for the missing impact of impact studies: no accountability with internal learning, no developed scaling out, the overlap of monitoring and evaluation and impact assessment, the intrinsic nature of functional and empowering farmer participation, the persistent lack of widespread attention to gender, and the operational and political complexity of multistakeholder impact assessment. In contrast, a desired impact of research could be reached or boosted by specific measures without making an impact assessment at all. Kristjanson,et al. (66), for example, proposed seven framework conditions for agricultural research to bridge the gap between scientific knowledge and action towards sustainable development. RIA should develop into processoriented evaluations, in contrast to outcome-oriented evaluation (67), for addressing the intended kind of impacts, the scope of assessment, and for choosing the appropriate assessment method (19).

Conclusion
This review aimed at providing an overview of impact assessment activities reported in academic agricultural literature with regard to their coverage of impact areas and type of assessment method used. We found a remarkable body of non-scientometric RIA at all evaluation levels of agricultural research but a major interest in economic impacts of new agricultural technologies. These are closely followed by an interest in social impacts at multiple assessments levels that usually focus on food security and poverty reduction and rely slightly more on qualitative assessment methods. In contrast, the assessment of the environmental impacts of agricultural research or comprehensive sustainability assessments was exceptionally limited. They may have been systematically overlooked in the past, for the reason of expected negative results, thought to be covered by other impact studies or methodological challenges. RIA could learn from user-oriented policy impact assessments that usually include environmental impacts. Frameworks for RIA should avoid narrowing the assessment focus and instead considering intended and unintended impacts in several impact areas equally. It seems fruitful to invest in assessment teams' environmental analytic skills and to expand several of the already developed methods for economic or social impact to the environmental impacts. Only then, the complex and comprehensive contribution of agricultural research to sustainable development can be revealed.