https://orcid.org/0000-0002-0997-3605
Abstract
Impact assessments in grant applications can be biased, random, or inconsistent. One reason is that there is not a framework to assist the review process. To develop fair and transparent evaluative criteria, it is necessary to understand what kinds of outcome and impact can reasonably be achieved and expected. Using content analysis, 100 UK Research Excellence Framework (REF) 2014 impact case studies were analysed based on the definition of the output, outcome, and impact in the logic model. The analysis shows that (1) outcomes and medium-term impacts, not long-term impacts, are reported in the case studies, (2) impacts can be categorised as use and experience based, and (3) there is a need to recognise the creation and co-creation of impacts. A framework for impact evaluation in grant applications has been developed. In this framework, the criteria for evaluating impact statements focus on process-oriented impacts and that ‘impacts’ can be outputs, outcomes, and medium-term impacts.
1. Introduction
Impact assessment is challenging. It is because the notion of impact encompasses a wide range of events and changes: some smaller in scale, some wider in scope; some focus on local communities, some aim at solving global challenges; some can be achieved tomorrow, while others may take decades or generations to be realised. Impact assessment is challenging also because individuals, from researchers to policymakers to the general public, have different conceptions of impact: what constitute positive impacts and meaningful contributions now and in the future? Is it more important to tackle homelessness or to boost the innovation economy? And is it more important for our children to be able to appreciate the beauty of literature and music and arts or to start coding at a young age? The choices are based on what we perceive as good and important as an individual and as a society. Collectively, these choices will affect where we put our limited resources in research funding and research support services. Impact assessment is challenging because it is often not clear as to what kinds of impact are expected and deemed valuable amongst the wide and varied, tangible and intangible societal benefits derived from the pursuit of knowledge.
Writing impact statements is often a taxing exercise. Isn’t the pursuit of knowledge impactful in-and-of itself? Why is it necessary to guess or predict impact in a grant application? And why is it necessary to prove the ‘value’ of research and scholarship in an impact case study? It is no surprise that impact claims can be ‘potentially thwarted, minimalised and/or fictionalised’ in impact case studies (Watermeyer 2014) and exaggerated in grant applications (Chubb and Watermeyer 2017). Writing impact statements is an onerous task because impact assessments do not provide clear guidance as to the types of impact expected (e.g. tangible or intangible, long term or medium term) and the criteria for which the desired impacts are evaluated upon.
If the purpose of impact assessments is to inculcate a research culture of responsible societal impact and to counter the academic misconduct and malpractices carried by the metrics tide (see Biagioli and Lippman 2020; Wilsdon et al. 2015), then what improvements can be made to make impact assessment a meaningful and fruitful exercise? And how can impact assessment be inclusive of the manifold normal contributions of research and scholarship (Sivertsen and Meijer 2020)? For many funding programmes, grant applicants are required to include an impact statement of their proposed research. However, the guidelines provided in call documents tend to define impact as a broad concept but without clear indications of which aspects of impact (e.g. the planning of impact activities or the predicted impacts) will be evaluated upon. Likewise, grant reviewers are not provided with a scoring scheme, and, as a result, their evaluation can be based on their own conception of impact and social good, as well as the funding agencies’ expected outcomes. To improve the fairness, consistency, and transparency of impact evaluation in grant applications and to encourage impact creation and co-creation of societal impact, it is necessary, first, to delineate as to what kinds of impact can be assessed—that is to say, instead of an all-encompassing definition of impact, there is a need to create finer categories that align with the objectives of a funding programme in ex ante impact assessment (Ma et al. 2020). Second, it is essential that impact is not perceived as yet another burdensome and meaningless tick-box exercise placed upon researchers and scholars, but rather, something that can be incorporated and manifested throughout the research process. And finally, impact assessment criteria should be created in accordance with the types of impact being evaluated to avoid biased and random judgements.
The aim of this study is to develop a framework for impact evaluation in grant applications for funding programmes that aim to support basic and curiosity-driven research, societal and global challenges, academic–industry or academic–non-governmental organisation (NGO) collaboration, and public engagement and science communication. The objectives are twofold: to understand ‘what impact is’ and what kinds of impact can be reasonably achieved within a certain time frame and to examine the processes and beneficiaries of impact activities for developing the criteria for ex ante impact assessment. By deconstructing impact reported in impact case studies, this study illustrates the ways by which researchers and scholars demonstrate the societal impact of their research and to demonstrate the kinds of impact that can be achieved in the short term, medium term, and longer term. Based on preliminary data analysis, a typology of use- and experience-based outcomes and impacts is created as the basis for the framework of impact assessment. In other words, the study first examines what impact is—not the definitions of impact, but what researchers had actually done and achieved—and then considers what can be assessed in the evaluation of impact statement in grant applications.
The rest of the article is structured as follows: the literature review consists of three parts—pathways to impact, impact as change/process, and challenges of impact assessment—to illustrate the need to examine ‘impacts’ based on the Kellogg’s Logic Model (hereafter ‘logic model’). The method Section 2 describes the approaches taken to analyse and categorise the selected UK REF2014 impact case studies. The findings and discussion Section 3 shows that the contents of impact case studies are mainly outcomes and medium-term impacts, and they are further categorised as use- and experience-based outcomes and impacts; the section also reflects on the importance of process-oriented impacts and the ambiguity of beneficiaries. Finally, a framework of evaluating impact is developed based on the findings.
1.1 Pathways to impact
The logic model of impact (also known as the linear model, see Penfield et al. 2014) is based on the original Logic Model Development Guide published by the Kellogg Foundation (2004):
Outputs are direct products of programme activities and may include types, levels, and targets of services to be delivered by the programme.
Outcomes are the specific changes in programme participants’ behaviour, knowledge, skills, status, and level of functioning.
Impact is fundamental intended or unintended change occurring in organizations, communities, or systems as a result of programme activities within 7–10 years.
Often, the logic model is illustrated as ‘impact journey’ where a research project may follow, including five stages: inputs, activities, outputs, outcomes, and impacts (Fig. 1). Similar to the Payback Framework (Donovan and Hanney 2011), the first three stages are mainly concerned with research activities, including the production of research outputs such as scholarly publications, databases, prototypes, and exhibitions and performances, whereas the outcome stage indicates the uptake of research evidenced by citations and awareness in non-academic publications and media coverage, as well as the use of devices, tools, instruments, and therapies. These outcomes are considered as steps towards impacts—however, they are not ‘impact’ by definition and hence, presumably, would not be evaluated favourably in impact assessments. The last stage of the impact journey indicates various categories of impact: academic, cultural, economic, educational, environmental, health, political, societal, and technological. Unlike the previous stages, there are no concrete examples that illustrate what impacts might look like and how impacts can be demonstrated, evidenced, or evaluated in the impact journey.
The impact journey (adapted from UCD Research Impact Toolkit—https://www.uck.ie/impacttoolkit/whatisimpact).
However, there is little evidence that impact assessment has strictly followed the definition or progression of impact in the logic model. The samples included in the Guidance of UK REF2021, for example, show that outcomes such as ‘A new diagnostic or clinical technology has been adopted’ or ‘Research is used by parliamentarians to develop proposals for new legislation’ can be included in an impact case study, although they are categorised as outcomes in the logic model. In an analysis of sixty impact case studies in the social sciences and humanities (SSH) from sixteen European countries, Muhonen et al. (2019) develop a typology of SSH pathways to societal impacts, including ‘The Classical Pipeline’ pathway (i.e. logic model), as well as non-linear pathways that are supported by various conditions such as unexpected political events or natural catastrophes. Societal impacts are achieved through dissemination, co-creation, reaction to societal change, and driving societal change. There is no distinction between outputs, outcomes, and impacts in their study.
For impact statements in grant applications, Ma et al. (2020) find that reviewers gave opinions about research outputs and commercial outcomes but only made cursory comments on long-term impacts such as ‘solving major national and global problems and challenges’. In the European Union (EU) Commission Impact Assessment working document, societal impacts are predetermined in accordance with the aims and objectives of the funding programme: (1) addressing EU policy priorities through Research and Innovation (R&I), (2) delivering benefits and impact through R&I missions, and (3) strengthening the uptake of innovation in society. These societal impacts are to be achieved by the production of short-term outputs that create medium-term solutions and then generate longer-term benefits for societal impacts. In this working document, the longer-term benefits are ‘outcomes’ in the logic model, which indicates uptake and use of research.
Over the years, it has also been suggested that impact assessment should focus on processes and contributions instead of outcome-oriented impacts because of the difficulties in evidencing and proving societal impacts. Spaapen and van Drooge (2011) propose the notion of productive interactions, defined as ‘exchanges between researchers and stakeholders in which knowledge is produced and valued that is both scientifically robust and socially relevant’ including direct, indirect, and financial interactions. Molas-Gallart and Tang (2011) maintain that the productive interactions approach will benefit the study of societal impact by tracing forward the activities that generate societal impacts. Similarly, contribution mapping has been proposed by Kok and Schuit (2012) as ‘a novel approach to research monitoring and evaluation that focuses on processes and contributions’ instead of products and impacts. The processes and contributions can include engaging policymakers, writing dissemination plans, and engaging patients in the interpretation of results. Ma et al. (2020) suggest that it is more appropriate for ex ante impact assessments to focus on process-oriented, rather than outcome-oriented, impacts, as it is often impossible for researchers and scholars to predict and evidence the long-term societal impacts of a research project at its proposal stage.
The logic model of impact can serve as a template for planning impact activities. In reality, however, the generation of societal impact takes many different pathways. However, so far, we know little about these processes. It is likely because the impact case studies method only assesses ‘reach and significance’ and not the processes that lead to societal impacts, while most reviews of impact statements in grant applications emphasize outcome-oriented impacts. Together, impact assessments have not served to illuminate the pathways and productive interactions that are beneficial for planning impact activities. There is also a discrepancy between the definition of impact in theory and in practice.
1.2 Impact as change, impact as process
Impact assessment is not possible without a working definition of societal impact. The UK REF2021 states that:
‘Impact includes, but is not limited to, an effect on, change or benefit to:
the activity, attitude, awareness, behaviour, capacity, opportunity, performance, policy, practice, process or understanding
of an audience, beneficiary, community, constituency, organisation, or individuals
in any geographic location whether locally, regionally, nationally or internationally.’
The scope of societal impact is wide and varied, although it is unclear as to how an improved awareness of cultural heritage in a small rural community would be weighed against the development of a vaccine for global use in the scoring scheme of the impact case studies method. Nevertheless, there seems to be a general understanding that societal impact implies change—as ‘something that changes people’s lives’ or something that has ‘made a difference to the world’ (Samuel and Derrick 2015). In an analysis of 162 case studies in community health sciences, Greenhalgh and Fahy (2015) found that the case studies reported influences in guidelines, changes policy and practice, improved morbidity, and reduced mortality. However, the analysis shows that most ‘impacts’ reported are short term to medium term, and some would be considered as ‘outcomes’ instead of ‘impacts’ in the logic model. For example, the claim that research studies ‘prompted vigorous public debate’ can be understood as uptake of research (outcomes); however, the debate may not lead to any actual changes (impacts). Heyeres et al. (2019) show that ‘intermediate impacts, which may or may not develop into long-term societal impacts were reported’ in most of the case studies and, in fact, only four papers met the actual UK REF criteria for societal impact in their study.
Furthermore, studies about impact case studies show that the processes and interactions by which researchers and scholars achieve impacts are rarely mentioned, likely because they are not ‘counted’ as societal impacts in impact assessments. Some active efforts have been observed by Greenhalgh and Fahy (2015) such as strong and ongoing links with policymakers and clinicians, development and delivery of training packages, and so on. A recent study of knowledge translation shows that the processes of achieving medium-term societal impacts can take longer than five years, not to mention the high costs involved (Caves and Lueling 2021). Thus, the question remains as to whether the aim of impact assessment is to encourage the creation and co-creation of societal impacts. If so, it seems that process-oriented impact would be much more appropriate for the purpose. Or, if impact assessment aims to trace and track societal impacts that naturally and organically emerge from research, then it seems to be more appropriate to evaluate the process of societal impacts generation over time.
Furthermore, studies of impact case studies show that there is a need to provide contextual information to show how changes occurred and what additional impacts can be achieved through different channels such as media coverage and invited public talks (Brook 2018). More importantly, there should be an emphasis on normal contributions (Sivertsen and Meijer 2020) throughout the research process. Upton et al. (2014) argue that more emphasis on process-based approach can better incentivise researchers and scholars in the creation and co-creation of societal impacts.
The terms ‘impact’ and ‘societal impact’ are usually defined broadly to be inclusive of the contributions of research and scholarship in society. The wide and varied societal impacts can make impact assessments difficult for reviewers because they can have different perceptions of social good, not to mention sometimes reviewers’ interpretations of societal impact can be based on the objectives of a funding programme and, in some cases, overwhelmingly show preferences for tangible economic outcomes (see de Jong et al. 2016; Ma et al. 2020).
1.3 Challenges of impact assessment
Attribution, causality, the counterfactual argument (what would have happened without the intervention), and the time lag between output and impact have been discussed as the main issues and challenges of impact assessment (see European Science Foundation 2012; Penfield et al. 2014). King’s College and Digital Science (2015) identify 3,709 unique impact pathways in their analysis of the UK REF 2014 impact case studies, as well as a diverse group of beneficiaries including companies, students, children, patients, schools, communities, NHS, teachers, women, families, governments, workers, clinicians, businesses, clients, manufacturers, ministers, parents, pupils, policymakers, museums, engineers, consultants, journalists, writers, citizens, consumers, volunteers, councils, charities, curators, designers, farmers, lawyers, animals, banks, and unions. There is, however, no information as to whether the potential beneficiaries were co-creators of societal impacts, and whether the potential beneficiaries were identified by a research project from the start or serendipitous recipients of societal impacts.
The evidence presented in impact case studies often points to immediate outcomes rather than longer-term impacts. For example, Brook (2018) records number reached, who was reached, what people did during the event, and what people said as evidence of societal impacts in art research. Outcomes such as spin-out companies, patents, and licences have also been documented as evidence of societal impacts, including revenue created from product sales, industrial investment, and staff employment; notably, one of the most frequent type of societal impacts is public policy and parliamentary debate (King’s College and Digital Science 2015). In a recent study, Kousha et al. (2021) also show that online evidence provided in impact case studies tend to support outcomes, for example, citations in clinical guidelines and inclusion in news articles, online videos, government websites, and social media and blogs.
When asked to suggest what are needed and what count in impact case studies, peer reviewers were ambivalent in their judgement of public engagement as a form of societal impact (Watermeyer and Hedgecoe 2016). Yet, reviewers also show generous marking as ‘celebration of science’ and ‘encouragement to universities who may be less established’ (Derrick and Samuel 2018). For ex ante impact assessment, there have been reports of random and inconsistent evaluation (Chubb and Watermeyer 2017; NABI 2018), which focuses on short-term commercial outcomes (de Jong et al. 2016; Ma et al. 2020). There are also uncertainties surrounding the political purpose and the notion of social good in impact assessments, in addition to a general lack of expertise or consensus in the evaluation of impact (Derrick and Samuel 2018; Holbrook and Frodeman 2011).
The challenges of impact assessment—as well as the (pro)active measures to increase the visibility of societal impacts—are not only concerned with the attributes of societal impact, but also contextual factors including societal impact capacity (de Jong and Muhonen 2020), while workplace cultures, stakeholder groups, and political and financial reasons have also been identified (Samuel and Derrick 2015). These factors are concerned with the opportunities whereby researchers can initiate or plan meaningful interactions with external stakeholders to achieve societal impacts. Furthermore, the provision of training and institutional support can make significant differences to translate and transform academic research into societal benefits (Hughes et al. 2019; Roberts 2009). However, there remain concerns about researcher motivation (Gentry et al. 2019) and academic freedom in the impact agenda (Smith et al. 2011; Watermeyer 2016).
Nevertheless, impact assessment is far from fair and transparent when the purposes of evaluation and the interpretation of societal impact remain ambiguous. What kinds of societal impacts researchers and scholars can achieve within a certain time frame? What are the appropriate criteria for evaluating what kinds of impact in an ex ante impact assessment? To answer these questions, this study analyses societal impacts in the continuum of outputs, outcomes, and impacts in the logic model based on 100 UK REF2014 impact case studies, followed by a framework for impact evaluation in grant applications that delineates the kinds of societal impact that can and should be evaluated for various types of funding programmes.
2. Method
Content analysis is a method for identifying the themes and patterns in textual as well as graphical materials. In this study, it is used to identify the ‘impacts’ reported in impact case studies for a better understanding of the beneficiaries, processes, and outcomes of impact activities. In the first phase of the study, a coding scheme was developed based on the definitions delineated in the original Logic Model Development Guide published by the Kellogg Foundation (2004), while the description and examples of outputs, outcomes, and impacts in the UCD Impact Toolkit (Fig. 1) are also taken into account.
Based on the preliminary findings, a second coding scheme was devised to capture the manifestations of impact represented in the impact case studies, including the categories of use-based and experience-based outcomes and impacts and medium-term and long-term impact: (1) use-based outcomes and impacts are evidenced by adaptation and use, whereas experience-based outcomes and impacts indicate influences and broader understanding, usually without concrete evidence due to the nature of activities and research; (2) medium-term impacts include changes in the medium term, for example, adaptation in local and professional practices and public policies; however the long-term impact, that is, the impact on culture, environment, health, and so on, has been not yet realised.
The UK REF2014 impact case studies are publicly available under the categories of submitting institution, unit of assessment, summary impact type, research subject area, impact UK location, and impact global location. For the purpose of this study, the impact case studies of four main panels (unit of assessment) were browsed to identity panels that are inclusive of different impact types—cultural, economic, environmental, health, legal, political, and technological—for examining the potential differences in their beneficiaries, outcomes, and impacts. Two multidisciplinary panels—Panel C17 Geography, Environmental Studies and Archaeology and Panel D34 Art and Design: History, Practices and Theory—were selected for analysis. The case studies were randomly chosen from the two panels when balancing each impact type.
The content analysis of the impact case studies was conducted by two coders. During the first phase, the first coder read and categorised the contents of the impact case studies and a second coder reviewed the content analysis and made queries about ambiguous categories, most of which were due to the lack of evidence in support of the claims of outcomes or societal impacts. Afterwards, another round of coding was conducted using the second coding scheme—the matrix of use- and experience-based outcomes/impacts and medium/long-term impacts—derived from the preliminary findings. Each impact case study was carefully read, and the contents were categorised into outputs, outcomes, and impacts, with notes indicating beneficiaries and evidence of impacts. The coding process reached saturation with about fifty case studies, but a further fifty samples were coded to ensure that there were no major gaps in knowledge. In total, 100 case studies were coded and analysed. In addition, examples of societal impacts in Annex A of the UK REF2021 Guidance were also coded according to the same schema and compared with the results of the analysis of impact case studies (Table A.1).
3. Findings and discussion
The impact case studies submitted to the UK REF2014, to a certain extent, represent the understanding and perception of societal impacts by researchers, scholars, and universities, on the one hand, and the kinds of societal impacts they could create and had achieved, on the other. This study shows similar kinds of societal impact represented in previous content analyses (e.g. Brook 2018; Greenhalgh and Fahy 2015), aligned with the examples listed in the Guidance document of UK REF2021 Annex A. The analysis also shows that the impact case studies report both ‘outcomes’ and ‘impacts’ in the logic model and that some outcomes and impacts are evidenced by adaptation and use, while others are experience-based. Most impacts reported are medium-term impacts, in the sense that they do not indicate long-term influences in society but intermediate changes such as implementation of procedures (i.e. the success or impact of implementation not yet realised). In fact, these changes may not be recognised as ‘impact’ by some reviewers (Watermeyer and Hedgecoe 2016). Further, the processes or impact activities that are contributive to or constitutive of the creation and/or co-creation of impacts are seldom reported in the impact case studies and the understanding of beneficiaries is ambiguous.
3.1 Use-based and experience-based outcomes and impacts
Although societal impacts are often defined and understood as long-term impacts, this study shows that most ‘impacts’ reported in the impact case studies are outcomes and medium-term impacts. Some outcomes and impacts are use-based in the sense that evidence is provided to show references, citations, and changes in policies and practices in non-academic settings such as educational resources, TV documentaries, and parliamentary debates. Some outcomes and impacts are experience-based, in the sense that direct and causal relationships cannot be observed or evidenced signified by verbs such as influenced, informed, stimulates, as well as terms such as understanding and awareness (Table 1).
Use-based and experience-based outcomes and impacts.
| Outcomes | Impacts | |
|---|---|---|
| Use based | Uptake of research, for example:
| Changes in practices and policies, for example:
|
| Experience based | Indication of influences in specific contexts, for example:
| Broader understanding and awareness, for example:
|
| Outcomes | Impacts | |
|---|---|---|
| Use based | Uptake of research, for example:
| Changes in practices and policies, for example:
|
| Experience based | Indication of influences in specific contexts, for example:
| Broader understanding and awareness, for example:
|
Use-based and experience-based outcomes and impacts.
| Outcomes | Impacts | |
|---|---|---|
| Use based | Uptake of research, for example:
| Changes in practices and policies, for example:
|
| Experience based | Indication of influences in specific contexts, for example:
| Broader understanding and awareness, for example:
|
| Outcomes | Impacts | |
|---|---|---|
| Use based | Uptake of research, for example:
| Changes in practices and policies, for example:
|
| Experience based | Indication of influences in specific contexts, for example:
| Broader understanding and awareness, for example:
|
Why are the impact case studies mainly consisted of outcomes and medium-term impacts? One reason can be the time frame in which the impact case studies were submitted. That is to say, it was only possible for the researchers to provide evidence of outcomes and medium-term impacts in terms of reach and significance because the long-term impacts cannot be traced, tracked, evidenced, or simply have not been realised yet. In one case study, for example, the submission states that,
‘Given the recent timing of the interventions in Cape Town and other provinces, it is premature to measure the impact of the Ukwazana programme in terms of the number of lives saved or transformed sexual practices, but the very fact that Anova decided to replicate the programme is an indicator of the social receptivity and relevance of the interventions.’
The actual impact claim in the case study is that the research project had ‘influenced the ways in which Health4Men engaged with and worked with volunteer and outreach workers’. In other words, the impact claim is about the take-up of the research (outcomes), but its longer-term impacts can only be anticipated and the evidence to support long-term impacts was not available.
Some impact case studies such as the example above show clear paths to societal impacts. These case studies reflect research that had identified specific issues or challenges as objectives of their studies. Therefore, they were able to specify beneficiaries such as communities affected by adverse weather or health or socio-economic conditions, and many were also able to pinpoint the policies and practices they had changed. However, since the assessment of these impact case studies was only concerned with reach and significance, few impact case studies describe the ways by which the researchers collaborate, cooperate, or interact with stakeholders. In other words, the activities that led to influences or changes in public policy or attracted widespread media attention were not documented. As a result, we know little about the process of so-called knowledge transfer or knowledge translation from the reading and analysis of impact case studies. Nevertheless, it is postulated that research aiming to solve or resolve existing challenges, including policy-oriented research, is more likely to disseminate their research to relevant stakeholders and to seek opportunities to make actual changes.
For many, it seemed to be difficult to evidence direct, one-to-one relationship between research and outcomes/impacts, resulting in statements about informing, influencing, and stimulating public opinions, policy and parliamentary debates, and so on. These statements imply uptake and use of research and perhaps even change, although often without tangible evidence. We consider these statements as experience-based outcomes, for opinions and debates tend to occur in specific time frames and do not necessarily lead to actual changes. In other words, the ‘take-up’ of research might have changed some people’s mind, yet its long-term impacts are undetermined and sometimes impossible to trace, track, or monitor. But yet again, because only reach and significance ‘count’ in impact case studies, the processes that led to experience-based outcomes are not described—not only that we learn little about the meaningful interactions that were fruitful for achieving societal impacts, but we are also negligent of the time and resources required to create and co-create societal impacts.
3.2 From influences to change
Initially, the categorisation of outcomes, medium-term impacts, and long-term impacts was coded under the impact type (e.g. cultural, political, societal, and technological) declared on the impact case studies; this study does not observe significant differences between impact types. Impact claims such as public awareness and debates are common amongst economic, environmental, political, and societal impact types, whereas citations and references in non-academic channels are common under all impact types. At times, there are cases where the outcomes and impacts described and the impact type do not seem to match. For example, one case study about the development of a technique in animation was submitted under ‘Technological Impact’; however, the evidence provided was about an increase in jobs and turnover. Another case study about access to basic bank accounts was submitted under ‘Economic Impact’ although its evidence shows ‘the number of adults living in households without access to a bank account fell from two million to 890,000’, which would be more appropriately labelled as societal impact.
Yet, there is one distinctive feature of case studies submitted under ‘Cultural Impact’. These impact case studies tend to mention more about public engagement activities with supporting evidence such as the number of visitors, downloads, and testimonials, similar to those reported by Brook (2018), despite the fact that, presumably, these activities do not ‘count’ in terms of reach and significance. This finding confirms the oft-discussed issues and challenges of impact assessment, particularly the consideration of normal versus extraordinary impacts (Sivertsen and Meijer 2020), as the enrichment of understanding of the arts, cultures, and sciences is not necessarily explicitly referenced or cited but absorbed into the fabrics of human lives.
Consider, for example, one of the most important cultural artefacts in the 20th century, Star Wars. There is no question that the Star Wars enterprise created and continues to create jobs and revenues (economic impact), while also innovated and applied techniques in film-making and special effects (technological impact), not to mention Star Wars represents certain historical and political thoughts (cultural, societal, and political impacts), as well as human (or galactic) conditions (environmental impact). The societal impacts of Star Wars are numerous, but how to evidence such impacts in terms of outcomes and impacts except for the revenue generated every year, and perhaps the number of viewers? And more importantly, would Star Wars have been created if George Lucas was not exposed to and influenced by materials about history, politics, religion, and space? Would it be possible to trace all of his influences that led to the creation of Star Wars? One can argue that the most important cultural impacts are but tangible or explicit in the immediate instance.
Exhibitions, educational workshops, public lectures, and podcasts contribute to longer-term societal impacts. Some are curated to tackle immediate challenges, for example, to raise awareness of the climate crisis, which may in turn influence public opinions that support changes in environmental policy, while some are about discoveries—archaeological, anthropological, historical, or scientific—that open our minds to cultural and natural phenomena. There are also displays of new invention and creative performances that motivate explorations and experiments. These different influences, as discussed, are not necessarily easily evidenced but would be generally accepted as contributions to societal benefits. When societal impacts are defined as changes, it is important to recognise the accomplishments—normal contributions—that will collectively bring longer-term impacts to the society. There are many pathways identified (Muhonen et al. 2019) for extraordinary impacts; however, it may be more important to recognise the normal contributions that form the basis for innovation and invention, public responsibility, and civic debates.
Understandably, the influences and transformative powers of serendipitous encounters are impossible to be captured in impact case studies, but it is a conundrum that the impact case studies method does not assess the processes in creating and co-creating societal impacts. Public engagement and similar activities. Even if the purpose of impact assessment is solely for the need of accountability, the contributions to create societal impacts should be taken into account—or otherwise researchers and scholars cannot justify the time and resources to engage in societal impact activities or even to recognise its importance other than an evaluation exercise. Furthermore, the attainment of outcomes and impacts is not entirely due to the contributions and quality of research. For example, politicians and policymakers have their own agendas and interests despite strong scientific evidence of climate change and evidence-based research about homelessness and poverty.
3.3 Whose benefits?
In the impact case studies analysed, the beneficiaries are clearly stated for research projects that aim to understand and resolve a specific challenge or issue, while most impact case studies either do not explicitly mention the beneficiaries or they indicate a relatively broad category such as ‘the public’. As discussed, when the objectives of a research project are to tackle a societal challenge in a region or to support a population of certain educational, health, socio-economic conditions and needs, the beneficiaries are pre-determined and hence easy to identify. When a research project leads to public policy debates, however, it cannot claim the public or the policymakers as ‘beneficiaries’ for long-term societal impacts because actual changes had not been realised based on the assessment criteria. Broadly speaking, the beneficiaries of use-based outcomes and impacts are easier to identify, whereas those of experience-based outcomes and impacts are wide and far-reaching but less specific.
One can argue that if the public and policymakers have been informed by a piece of research, they should be counted as beneficiaries—however that would require a rethinking and redesign of impact assessment to identify the intermediate recipients or beneficiaries as a result of knowledge production and dissemination. Consider a typical example of impact statement in a case study:
‘The exhibition had an impact on multiple audiences as substantiated by the large visitor numbers, the success of the schools’ programme, community outreach workshops, curatorial tours and study days, the subsequent interest in the use of these experimental techniques by museum/academic institutions overseas, and extensive coverage internationally in the traditional media and on the internet.’
The statement presents successful experience-based outcomes of a research project, although the outcomes and beneficiaries are not specific. It is clear that the researcher(s) had devoted significant time and energy in public engagement and outreach, including exhibition, schools’ programme, community workshops, and curatorial tours, but the long-term societal impacts remain unknown while these activities can only be, if, counted as take-up of research. Strictly speaking, schools, local communities, and museum/academic institutions acted as intermediaries in the dissemination of information and knowledge but not beneficiaries because it is unclear whether the experimental techniques mentioned will actually change practices and subsequently how we live and learn and so on.
Again, since the UK REF impact assessment does not give credits to the processes of creating societal impacts, few impact case studies describe how the researchers collaborate and reach out to policymakers, mass media, and other stakeholders. It is unclear as to whether there were existing relations or the research has been ‘taken up’ without any intervention or effort. There is also no distinction between intermediaries, beneficiaries, and co-creators of knowledge production and dissemination. For research projects that lead to the longer-term, experience-based impacts, their beneficiaries can be very broad; however, the causal relationship between a research project and a person’s everyday actions cannot be specified or identified.
Are use-based outcomes and impacts more beneficial than experience-based ones? The question is analogous to one that asks if we should pursue knowledge if it does not have immediate instrumental uses. However, both questions fail to recognise that research projects that aim to provide immediate instrumental use are built on the work of others, including research and scholarship that may seem to be not instrumental at the time of its discovery or creation. While it is important to bolster research that tackles global and local challenges, it is also essential to recognise the contributions that will lead to greater public understanding and awareness (experience-based outcomes and impacts), not to mention that these contributions are building blocks of research that aim to provide solutions at times of crisis such as the development of vaccines. There is a need to rethink, then, beneficiaries at different stages of the impact journey. For some research projects, experience-based outcomes can be created by public engagements and science communication, and their longer-term impacts would be based on the collective contributions to the knowledge base. Local communities, schools, and cultural institutions can benefit from impact activities although their impacts are unknown in a near future.
3.4 Summary of findings
The findings of this study show that societal impacts described in impact case studies are mainly outcomes and medium-term impacts. Outcomes are defined in the logic model as ‘take-up’ of research; medium-term impacts implicate changes in policies and practices; however, their long-term impacts are unknown or not yet realised. Some outcomes and impacts are use-based, meaning that uptake of research had been documented, whereas some outcomes and impacts are experience-based, usually qualified by influences and stimulation. These outcomes and medium-term impacts were the achievements, and sometimes, activities, researchers, and scholars can accomplish and have achieved within a certain time frame. Furthermore, the beneficiaries are not commonly stated in impact case studies and there are uncertainties as to who can be counted as beneficiaries, for example, those who benefit from the longer-term societal impacts or those who are informed in public engagement and science communication activities.
The reason that outcomes and medium-term impacts—rather than long-term impacts—were described in impact case studies can be considered with the challenges and issues of impact assessment in attribution and causality, for it is more feasible to indicate uptake and translation of research or impact activities that are traceable to the research projects or published works. In other words, it is possible to state what one had contributed to the uptake of public engagement events; however, it is difficult if not impossible to attribute longer-term impacts to a specific person or a research project. These findings have implications for future uses of the impact case studies method as societal impacts not only encompass a wide range of impact types but also short- to longer-term outcomes and impacts. That is to say, any impact assessment should consider whether outcomes, medium-term impacts, or longer-term impacts would be evaluated and, more importantly, would process-oriented impacts including impact activities such as public engagement and science communication be ‘counted’. Hence, there are many considerations and complications in devising ex post impact assessments. The impact case studies method, to a certain extent, is a one-size-fits-all model which can result in generous marking (Derrick and Samuel 2018), meaning that the assessment can be unfair when it comes to allocation of resources, and the impact case studies method surely cannot be used for evaluating individuals’ performance with the rather loose evaluative criteria when societal impacts can be outcomes, medium-term impacts, and longer-term impacts and when public engagement and science communication activities are not valued or rewarded.
The logic model of impact is useful for differentiating the various stages towards impact, albeit the actual pathways can be convoluted and complex. Hence, it can be used at the planning phase of a resource project with the understanding that the take-up of research and expected impacts are sometimes dependent on factors that cannot be controlled by the researchers. Furthermore, the findings of this study provide a better understanding of what can be realistically accomplished and expected, that is, instead of predictions and guesses of long-term societal impacts in grant applications. In the following, we propose a framework for ex ante impact assessments.
3.5 A framework for evaluating impact statements in grant applications
This study shows that outcomes and medium-term impacts are most frequently reported in impact case studies, meaning that long-term impact cannot be expected during a grant period and is not necessary and even inappropriate to be assessed in grant applications. The study also shows that a one-size-fits-all concept of impact is not useful for evaluating impact statements, especially if the aim of a funding programmes is to stimulate immediate and tangible outputs and outcomes. That is to say, expected ‘impacts’ can be outputs, outcomes, and impacts in the logic model. Based on the findings about use- and experience-based outcomes and impacts, as well as the importance of impact activities, an assessment framework for the following four main types of funding programme is considered: basic, societal challenges, collaboration, and publication engagement (Table 2).
A framework for evaluating impact in grant applications.
| Type of funding programme | Basic | Societal challenges | Collaboration | Public engagement |
|---|---|---|---|---|
| Aim of funding programmes | Support basic and curiosity-driven research | Address societal challenges, including Sustainable Development Goals (SDGs) and/or national priorities | Encourage collaboration between academia and industry, NGOs, and other stakeholders | Enrich cultural understanding and experience and/or promote science communication |
| Impact assessment? | No | Yes | Yes | Yes |
| What to assess? | N/A | The societal challenges and national priorities the research project aims to address; the beneficiaries, practices, or policies the research project aims to inform, influence, or change | The outputs, expected use-based outcomes of the collaboration; expected experience-based outcomes and impacts can also be included; the potential of longer-term collaboration; the societal challenges and national priorities the collaboration aims to address | The processes and activities the research project plans to undertake; the expected experience-based outcomes of the activities; the collaborators and participants of the proposed activities |
| Type of funding programme | Basic | Societal challenges | Collaboration | Public engagement |
|---|---|---|---|---|
| Aim of funding programmes | Support basic and curiosity-driven research | Address societal challenges, including Sustainable Development Goals (SDGs) and/or national priorities | Encourage collaboration between academia and industry, NGOs, and other stakeholders | Enrich cultural understanding and experience and/or promote science communication |
| Impact assessment? | No | Yes | Yes | Yes |
| What to assess? | N/A | The societal challenges and national priorities the research project aims to address; the beneficiaries, practices, or policies the research project aims to inform, influence, or change | The outputs, expected use-based outcomes of the collaboration; expected experience-based outcomes and impacts can also be included; the potential of longer-term collaboration; the societal challenges and national priorities the collaboration aims to address | The processes and activities the research project plans to undertake; the expected experience-based outcomes of the activities; the collaborators and participants of the proposed activities |
A framework for evaluating impact in grant applications.
| Type of funding programme | Basic | Societal challenges | Collaboration | Public engagement |
|---|---|---|---|---|
| Aim of funding programmes | Support basic and curiosity-driven research | Address societal challenges, including Sustainable Development Goals (SDGs) and/or national priorities | Encourage collaboration between academia and industry, NGOs, and other stakeholders | Enrich cultural understanding and experience and/or promote science communication |
| Impact assessment? | No | Yes | Yes | Yes |
| What to assess? | N/A | The societal challenges and national priorities the research project aims to address; the beneficiaries, practices, or policies the research project aims to inform, influence, or change | The outputs, expected use-based outcomes of the collaboration; expected experience-based outcomes and impacts can also be included; the potential of longer-term collaboration; the societal challenges and national priorities the collaboration aims to address | The processes and activities the research project plans to undertake; the expected experience-based outcomes of the activities; the collaborators and participants of the proposed activities |
| Type of funding programme | Basic | Societal challenges | Collaboration | Public engagement |
|---|---|---|---|---|
| Aim of funding programmes | Support basic and curiosity-driven research | Address societal challenges, including Sustainable Development Goals (SDGs) and/or national priorities | Encourage collaboration between academia and industry, NGOs, and other stakeholders | Enrich cultural understanding and experience and/or promote science communication |
| Impact assessment? | No | Yes | Yes | Yes |
| What to assess? | N/A | The societal challenges and national priorities the research project aims to address; the beneficiaries, practices, or policies the research project aims to inform, influence, or change | The outputs, expected use-based outcomes of the collaboration; expected experience-based outcomes and impacts can also be included; the potential of longer-term collaboration; the societal challenges and national priorities the collaboration aims to address | The processes and activities the research project plans to undertake; the expected experience-based outcomes of the activities; the collaborators and participants of the proposed activities |
For funding programmes that support basic and curiosity-driven research, the purpose is to provide intellectual space and resources for exploring questions that are deemed important and challenging by the research community. Although these questions may not directly address a crisis or create a product, they may lead to more far-reaching societal impacts in the long term. It is hence inappropriate to predict actual societal impacts that a research project may generate—as predictions may be merely fictional or general statements, often presented in phrases such as ‘a deeper understanding of…’ in impact case studies. It is not to deny the validity of important long-term impacts; however, such statements cannot be assessed at the proposal stage because no evidence of societal impacts can be provided. It would also be counter-productive should the creation of societal impacts is placed more significance than the research itself. The high risk, high uncertainty nature of basic research does not lend itself for impact assessment in grant applications. The evaluation of grant application in basic, curiosity-driven research should be based on the quality of the research proposal agreed upon by experts in the research area.
For funding programmes that address global and societal challenges and/or national priorities, the long-term societal impacts have already been identified by the funding agencies. This study shows that research that aims to tackle existing societal challenges can identify specific beneficiaries, professional practices, and public policies that it intends to influence and change. As such, it would be appropriate for applicants to articulate which global or societal challenges the research projects will address and, where possible, provide examples of policy and practice the research project may change or influence. In other words, the grant application does not predict what actually will happen, but it can show its relevance to the funding programmes by identifying pathways whereby societal impacts may be created. Depending on the objectives of a funding programme, plans for impact creation and/or co-creation can be assessed. Yet, these activities should be secondary to the quality of the research design. Hence, there should also be consideration in terms of scoring—that is, if the impact statement should be scored, and if so, how much it would weigh overall.
Increasingly, there are funding programmes that aim to encourage collaboration between academic research and NGOs, and co-creation between academic and industry. These funding programmes are sometimes promoted as knowledge transfer or knowledge translation. The societal impacts of academic–NGO collaboration are primarily based on the missions of the organisation, for example, homelessness, poverty, and people with under-supported mental and physical health conditions. In other words, there are immediate societal challenges and issues to be resolved. The evaluation of impact statement in these instances should hence focus on the expected use-based and/or experience-based outcomes of the collaboration. For example, how will the NGOs make use of research in their work? Will they develop public engagement activities to raise public awareness and/or intervention in policy debates? How will the collaboration enhance the work of the NGOs and its missions? These questions are relevant to the shorter- and medium-term outcomes, and when appropriate, an impact statement can also articulate the potential longer-term impacts for a community.
For academic–industry collaboration, however, the goals can be relatively short-term, for example, the development of products, patents, licences, and so on. In other words, these collaborations may aim to produce ‘outputs’ in the logic model. For some projects, it is possible to articulate the longer-term impacts of products such as biodegradable plastics; others are not necessarily aimed at solving societal challenges but a commercial and/or innovative endeavour. Whether outcomes and impacts should be evaluated, then, would depend on the objectives of the funding programmes. That is to say, whether a programme aims to tackle societal challenges by encouraging academic–industry collaboration or to promote knowledge transfer for other purposes such as to increase the capacity of a commercial sector.
Lastly, funding programmes that encourage public engagement and science communication are largely aimed at enriching cultural experiences inasmuch as scientific and innovative aspirations. The topics and themes can range from arts, literature, history to mathematics and space. These activities encourage and inspire inquisitive minds to explore and invent, to ask questions, and to engage in civic debates. The long-term societal impacts and their beneficiaries are largely unknown and cannot be predicted, but, as this study shows, exhibitions and public talks, participating schools, and media relations can be demonstrated. In truth, these funding programmes have already identified the benefits of public engagement and science communication activities in-and-of themselves, for they are normal contributions that may lead to extraordinary societal impacts in the long term. Therefore, when evaluating impact statements for these programmes, it would be appropriate to assess the planned activities, the expected outcomes, as well as existing and potential collaborators and participants in the intermediate term. Although the long-term societal impacts and their beneficiaries cannot be predicted or evidenced, activities and plans to reach out to targeted audience via different channels can be assessed.
All in all, impact assessment in grant applications should align with the aims and objectives of the funding programmes (see also Ma et al. 2020). The typology of use- and experience-based outcomes and impacts can be mapped for specific goals in the development of impact assessment in grant applications. As the analysis of impact case studies suggests, there is no one-size-fits-all definition of societal impact in the context of impact assessment. Considering the many challenges and issues of evaluating long-term societal impacts, this study suggests that ‘impact’ should be assessed based on what research projects can realistically achieve within a certain time frame and can include outputs, outcomes, and impacts in the logic model.
4. Conclusion
The findings of this study contribute to the design and implementation of impact assessments because they show, in the case of ex ante impact assessments, what kinds of societal impact can be reasonably created and achieved, and in the case of ex post impact assessment, how normal contributions (Sivertsen and Meijer 2020) can be appraised in addition to extraordinary ones to inculcate a culture of responsible societal impacts (AUTHOR in press). Based on the analysis of impact case studies, we propose a framework for impact evaluation in grant applications. The framework can be used to differentiate and develop evaluative criteria in ex ante impact assessments for different types of funding programme. Clearer and more transparent evaluative criteria will reduce uncertainties for both grant applicants and reviewers and increase fairness and efficiency of impact assessments. Most importantly, the findings support the switch from outcome-oriented impacts to process-oriented impacts (Ma et al. 2020) so as to account for significant efforts in achieving societal impacts. One can plan and do our best and improve our approaches to achieve societal impacts, but these efforts may not come to fruition due to factors other than research or impact activities (e.g. de Jong and Muhonen 2020), not to mention not all societal impacts can be traced and recorded. For instance, historical research is not constrained in scholarly works, as they are often taken and represented in books, magazine articles, podcasts, TV programmes, films, and games. When the audience and critics scrutinize the costume, the things people use, the ways people talk, the mannerism, and the social classes in these cultural artefacts and events, they demonstrate the impact of historical research. After all, is it possible to know how bits and pieces of knowledge can transform people’s lives? Would it be possible or ethical or moral to ask how a novel transforms one’s understanding of race, gender, sexual violence, and many other things? Societal impact is a collaborative endeavour based on community values and public goods at a societal level. Impact assessments may not be the best means to achieve such outcomes.
In the future, it would be important to consider the potential unintended consequences if impact assessments are to be continued and become more prominent in research evaluation at various levels. The impact case studies method, for instance, can lead to goal displacement. As Watermeyer (2016) suggests, ‘A more profound concern is that in the emerging “impact economy,” producing the best research comes second to producing the best impact’. There is also a danger that the impact case studies method engenders competition, rather than collaboration, between researchers because researchers are asked to ‘claim’ societal impacts, which would limit the overall societal impacts and, in some cases, may stifle not only the progress of research and scholarship, but also contributions that are responsive to the society.
Funding
This study is partly supported by Science Foundation Ireland (17/SPR/5319).
Conflict of interest statement.
None declared.
References
—— (
Appendix
Analysis of outcomes, medium-term impacts, and long-term impacts based on examples in UK REF2021 Guidance Annex A.
| Areas of impact | Outcomes | Medium-term impacts | Long-term impacts |
|---|---|---|---|
| Impacts on health and well-being of people and animal welfare | A new diagnostic or clinical technology has been adopted | Outcomes for patients/users or related groups have improved | Public health or well-being has improved |
| Increased patient/user involvement in shaping and implementing policy and practice | Misleading health claims identified by research are not included in food packaging | Quality of life in a developing country has been improved by new products or processes | |
| Impacts on creativity, culture, and society | Co-production of new cultural artefacts, including, for example, films, novels, and TV programmes | Generating new ways of thinking that influence creative practice, its artistic quality, or its audience reach | Developing stimuli to cultural tourism and contributing to the quality of the tourist experience |
| Inspiring, co-creating, and supporting new forms of artistic, literary, linguistic, social, economic, religious, and other expression | Increased understanding of local traditions leads to enhanced cultural preservation in any given context, for example, developing countries | ||
| Impact on social welfare | Influential contributions to campaigns for social, economic, political, and/or legal change through engagement with civil society groups | Improved social welfare, equality, social inclusion; improved access to justice and other opportunities (including employment and education) | Research has contributed to community regeneration or development |
| Enhanced understanding of victims’ needs in reconciliation processes in post-conflict states | Changes to social policy have been informed by research | Improved social and educational inclusion of marginalised groups in any given context, for example, developing countries | |
| Impacts on commerce and the economy | A spin-out or new business has been created, established its viability, or generated revenue or profits | Gains in productivity have been realised as a result of research-led changes in practice | Contributing to economic prosperity via the creative sector including publishing, music, theatre, museums and galleries, film and television, fashion, tourism, and computer games |
| Contributing to innovation and entrepreneurial activity through the design and delivery of new products or services | Policies have been introduced, which have had an impact on economic growth or incentivising productivity | Better access to finance opportunities | |
| Impacts on public policy, law, and services | Policy debate has been stimulated or informed by research evidence, which may have led to confirmation of policy, change in policy direction, implementation, or withdrawal of policy | The quality, accessibility, acceptability, or cost-effectiveness of a public service has been improved | (Sections of) the public have benefited from public service improvements |
| Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence | The work of an NGO, charitable or other organisation has been influenced by the research | Risks to the security of nation states have been reduced | |
| Impacts on production | Research helps to create routes to international innovation and market impact | Production, yields, or quality have been enhanced or level of waste has been reduced | Research leads to improvement in productivity and resource-use efficiency |
| Animal husbandry methods have changed | |||
| Impacts on practitioners and delivery of professional services, enhanced performance, or ethical practice | Practitioners/professionals/lawyers have used research findings in conducting their work. The development of expert systems has been influenced in areas such as medicine, human resources, accounting, and financial services | Professional bodies and learned societies have used research to define best practice, formulate policy, or lobby government or other stakeholders | Contribution to continuing personal and professional development |
| Professionals and organisations are able to adapt to changing cultural values as a result of research | The quality, efficiency, or productivity of a professional service has improved |
| Areas of impact | Outcomes | Medium-term impacts | Long-term impacts |
|---|---|---|---|
| Impacts on health and well-being of people and animal welfare | A new diagnostic or clinical technology has been adopted | Outcomes for patients/users or related groups have improved | Public health or well-being has improved |
| Increased patient/user involvement in shaping and implementing policy and practice | Misleading health claims identified by research are not included in food packaging | Quality of life in a developing country has been improved by new products or processes | |
| Impacts on creativity, culture, and society | Co-production of new cultural artefacts, including, for example, films, novels, and TV programmes | Generating new ways of thinking that influence creative practice, its artistic quality, or its audience reach | Developing stimuli to cultural tourism and contributing to the quality of the tourist experience |
| Inspiring, co-creating, and supporting new forms of artistic, literary, linguistic, social, economic, religious, and other expression | Increased understanding of local traditions leads to enhanced cultural preservation in any given context, for example, developing countries | ||
| Impact on social welfare | Influential contributions to campaigns for social, economic, political, and/or legal change through engagement with civil society groups | Improved social welfare, equality, social inclusion; improved access to justice and other opportunities (including employment and education) | Research has contributed to community regeneration or development |
| Enhanced understanding of victims’ needs in reconciliation processes in post-conflict states | Changes to social policy have been informed by research | Improved social and educational inclusion of marginalised groups in any given context, for example, developing countries | |
| Impacts on commerce and the economy | A spin-out or new business has been created, established its viability, or generated revenue or profits | Gains in productivity have been realised as a result of research-led changes in practice | Contributing to economic prosperity via the creative sector including publishing, music, theatre, museums and galleries, film and television, fashion, tourism, and computer games |
| Contributing to innovation and entrepreneurial activity through the design and delivery of new products or services | Policies have been introduced, which have had an impact on economic growth or incentivising productivity | Better access to finance opportunities | |
| Impacts on public policy, law, and services | Policy debate has been stimulated or informed by research evidence, which may have led to confirmation of policy, change in policy direction, implementation, or withdrawal of policy | The quality, accessibility, acceptability, or cost-effectiveness of a public service has been improved | (Sections of) the public have benefited from public service improvements |
| Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence | The work of an NGO, charitable or other organisation has been influenced by the research | Risks to the security of nation states have been reduced | |
| Impacts on production | Research helps to create routes to international innovation and market impact | Production, yields, or quality have been enhanced or level of waste has been reduced | Research leads to improvement in productivity and resource-use efficiency |
| Animal husbandry methods have changed | |||
| Impacts on practitioners and delivery of professional services, enhanced performance, or ethical practice | Practitioners/professionals/lawyers have used research findings in conducting their work. The development of expert systems has been influenced in areas such as medicine, human resources, accounting, and financial services | Professional bodies and learned societies have used research to define best practice, formulate policy, or lobby government or other stakeholders | Contribution to continuing personal and professional development |
| Professionals and organisations are able to adapt to changing cultural values as a result of research | The quality, efficiency, or productivity of a professional service has improved |
Analysis of outcomes, medium-term impacts, and long-term impacts based on examples in UK REF2021 Guidance Annex A.
| Areas of impact | Outcomes | Medium-term impacts | Long-term impacts |
|---|---|---|---|
| Impacts on health and well-being of people and animal welfare | A new diagnostic or clinical technology has been adopted | Outcomes for patients/users or related groups have improved | Public health or well-being has improved |
| Increased patient/user involvement in shaping and implementing policy and practice | Misleading health claims identified by research are not included in food packaging | Quality of life in a developing country has been improved by new products or processes | |
| Impacts on creativity, culture, and society | Co-production of new cultural artefacts, including, for example, films, novels, and TV programmes | Generating new ways of thinking that influence creative practice, its artistic quality, or its audience reach | Developing stimuli to cultural tourism and contributing to the quality of the tourist experience |
| Inspiring, co-creating, and supporting new forms of artistic, literary, linguistic, social, economic, religious, and other expression | Increased understanding of local traditions leads to enhanced cultural preservation in any given context, for example, developing countries | ||
| Impact on social welfare | Influential contributions to campaigns for social, economic, political, and/or legal change through engagement with civil society groups | Improved social welfare, equality, social inclusion; improved access to justice and other opportunities (including employment and education) | Research has contributed to community regeneration or development |
| Enhanced understanding of victims’ needs in reconciliation processes in post-conflict states | Changes to social policy have been informed by research | Improved social and educational inclusion of marginalised groups in any given context, for example, developing countries | |
| Impacts on commerce and the economy | A spin-out or new business has been created, established its viability, or generated revenue or profits | Gains in productivity have been realised as a result of research-led changes in practice | Contributing to economic prosperity via the creative sector including publishing, music, theatre, museums and galleries, film and television, fashion, tourism, and computer games |
| Contributing to innovation and entrepreneurial activity through the design and delivery of new products or services | Policies have been introduced, which have had an impact on economic growth or incentivising productivity | Better access to finance opportunities | |
| Impacts on public policy, law, and services | Policy debate has been stimulated or informed by research evidence, which may have led to confirmation of policy, change in policy direction, implementation, or withdrawal of policy | The quality, accessibility, acceptability, or cost-effectiveness of a public service has been improved | (Sections of) the public have benefited from public service improvements |
| Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence | The work of an NGO, charitable or other organisation has been influenced by the research | Risks to the security of nation states have been reduced | |
| Impacts on production | Research helps to create routes to international innovation and market impact | Production, yields, or quality have been enhanced or level of waste has been reduced | Research leads to improvement in productivity and resource-use efficiency |
| Animal husbandry methods have changed | |||
| Impacts on practitioners and delivery of professional services, enhanced performance, or ethical practice | Practitioners/professionals/lawyers have used research findings in conducting their work. The development of expert systems has been influenced in areas such as medicine, human resources, accounting, and financial services | Professional bodies and learned societies have used research to define best practice, formulate policy, or lobby government or other stakeholders | Contribution to continuing personal and professional development |
| Professionals and organisations are able to adapt to changing cultural values as a result of research | The quality, efficiency, or productivity of a professional service has improved |
| Areas of impact | Outcomes | Medium-term impacts | Long-term impacts |
|---|---|---|---|
| Impacts on health and well-being of people and animal welfare | A new diagnostic or clinical technology has been adopted | Outcomes for patients/users or related groups have improved | Public health or well-being has improved |
| Increased patient/user involvement in shaping and implementing policy and practice | Misleading health claims identified by research are not included in food packaging | Quality of life in a developing country has been improved by new products or processes | |
| Impacts on creativity, culture, and society | Co-production of new cultural artefacts, including, for example, films, novels, and TV programmes | Generating new ways of thinking that influence creative practice, its artistic quality, or its audience reach | Developing stimuli to cultural tourism and contributing to the quality of the tourist experience |
| Inspiring, co-creating, and supporting new forms of artistic, literary, linguistic, social, economic, religious, and other expression | Increased understanding of local traditions leads to enhanced cultural preservation in any given context, for example, developing countries | ||
| Impact on social welfare | Influential contributions to campaigns for social, economic, political, and/or legal change through engagement with civil society groups | Improved social welfare, equality, social inclusion; improved access to justice and other opportunities (including employment and education) | Research has contributed to community regeneration or development |
| Enhanced understanding of victims’ needs in reconciliation processes in post-conflict states | Changes to social policy have been informed by research | Improved social and educational inclusion of marginalised groups in any given context, for example, developing countries | |
| Impacts on commerce and the economy | A spin-out or new business has been created, established its viability, or generated revenue or profits | Gains in productivity have been realised as a result of research-led changes in practice | Contributing to economic prosperity via the creative sector including publishing, music, theatre, museums and galleries, film and television, fashion, tourism, and computer games |
| Contributing to innovation and entrepreneurial activity through the design and delivery of new products or services | Policies have been introduced, which have had an impact on economic growth or incentivising productivity | Better access to finance opportunities | |
| Impacts on public policy, law, and services | Policy debate has been stimulated or informed by research evidence, which may have led to confirmation of policy, change in policy direction, implementation, or withdrawal of policy | The quality, accessibility, acceptability, or cost-effectiveness of a public service has been improved | (Sections of) the public have benefited from public service improvements |
| Policy decisions or changes to legislation, regulations or guidelines have been informed by research evidence | The work of an NGO, charitable or other organisation has been influenced by the research | Risks to the security of nation states have been reduced | |
| Impacts on production | Research helps to create routes to international innovation and market impact | Production, yields, or quality have been enhanced or level of waste has been reduced | Research leads to improvement in productivity and resource-use efficiency |
| Animal husbandry methods have changed | |||
| Impacts on practitioners and delivery of professional services, enhanced performance, or ethical practice | Practitioners/professionals/lawyers have used research findings in conducting their work. The development of expert systems has been influenced in areas such as medicine, human resources, accounting, and financial services | Professional bodies and learned societies have used research to define best practice, formulate policy, or lobby government or other stakeholders | Contribution to continuing personal and professional development |
| Professionals and organisations are able to adapt to changing cultural values as a result of research | The quality, efficiency, or productivity of a professional service has improved |
