Normal versus extraordinary societal impact: how to understand, evaluate, and improve research activities in their relations to society?

Societal impact of research does not occur primarily as unexpected extraordinary incidents of particularly useful breakthroughs in science. It is more often a result of normal everyday interactions between organizations that need to create, exchange, and make use of new knowledge to further their goals. We use the distinctions between normal and extraordinary societal impact and between organizationaland individual-level activities and responsibilities to discuss how science–society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research.


Introduction
Societal impact has gained a central focus in research policy and evaluation. Research is increasingly expected to meet societal challenges and to interact responsibly with society. According to Greenhalgh et al. (2016) 'Impact occurs when research generates benefits (health, economic, cultural) in addition to building the academic knowledge base'. According to the European Commission's Expert Group on Knowledge Transfer Indicators, societal impact is comparable with knowledge transfer. Thus, they state that 'knowledge transfer encompasses all functions that may lead to improved use of knowledge development and held in the research sector for the benefit of society and its individuals' (Finne et al. 2011). This is one example of how societal impact has come to encompass many different terms or meanings. 'Third stream activities, societal benefits, societal quality, usefulness, public values, knowledge transfer and societal relevance' (Bornmann 2013) are other examples. National and international research funding organizations are asking for evidence or indicators of societal impact, and several frameworks for the understanding and evaluation of societal impact have already been proposed and piloted. In a study into research impact conducted by Greenhalgh et al. (2016), more than 20 existing models and frameworks were referenced, among them the SIAMPI productive interaction model (Molas-Gallart and Tang 2011; Spaapen and van Drooge, 2011;Jong et al. 2014) and the UK Research Excellence Framework (REF2014 2012). SIAMPI means 'Social Impact Assessment Methods for research and funding instruments through the study of Productive Interactions between science and society'. An updated overview of methods and frameworks for societal impact evaluation is given by Pedersen et al. (2019). Our purpose is to propose how evaluation and policy designs can be improved and made more relevant and effective using a distinction between normal and extraordinary societal impact and by separating between organizational-and individual-level activities and responsibilities in science-society relations. We will immediately use two examples from archeology and two examples from medical research to illustrate the usefulness of the two distinctions.
The Swiss traveler, geographer, and orientalist Johann Ludwig Burckhardt (1784-1817) is famous for his rediscovery in 1812 of the ancient Jordanian city of Petra, a UNESCO World Heritage Site since 1985. According to his own account, he was no more than the first Westerner to be introduced to the hidden Nabataean city. The natives had already been excavating and preserving the site as part of their societal life. The extraordinary impact of Burckhardt as an individual in the history of archeology was in fact based on the collective societal enterprise that archeology normally is. Today, investigations of the ancient Mediterranean civilizations are organized as large projects that are agreed with national authorities, funded from several countries, and performed in international collaboration. One example is the Norwegian-Syrian collaboration on documenting the ancient UNESCO heritage of the archeological site of Palmyra in Syria. It started in 2008 and proceeded as a normal international archeological project until the outbreak of the civil war in 2011. Then, Palmyra was bombed by the Islamic State (IS). The Syrian director of the site was beheaded while the Norwegian experts escaped. The documentation work done before the destruction now gained immense societal value and international interest. In 2016, the Norwegian scientists could report the extraordinary impact of their work to a national evaluation of societal impact in the humanities. Had Palmyra not been destructed, their work would have had the normal societal impact of the humanities that is often taken for granted and seldom receives attention.
Medical research organizations need patients for research, but they also serve these patients as part of a country's health-care system. Our example of societal impact in medical research is a negative one, the so-called Macchiarini case at the Karolinska Institute in Stockholm, where scientific fraud and clinical misconduct resulted in the death of patients (Nature 2016). Karolinska Institute first disregarded various allegations against their scientist. They then chose to treat the case as an extraordinary exception and individual responsibility. In our perspective, this way of thinking is the counterpart to only looking for extraordinary successful contributions from individual scientists. The Karolinska Institute finally took the responsibility as an organization and recognized that the Macchiarini case was a violation of the normal 'societal contract' between Swedish medical research organizations and Swedish society. Our second example from medical research indicates that it is normal to regard societal impact as an organizational responsibility on the user's side, in health-care organizations. The so-called Health Technology Assessment (HTA), which is practiced in the health-care sector of many countries, is a well-organized methodology to responsibly deal with normal relations between research and healthcare practices ).

The distinctions and their definitions
We define normal societal impact as the results of active, productive, and responsible interactions between (units of) research organizations and other organizations according to their purposes and aims in society. Within the research organizations, such interactions will often occur informally at the individual researcher or research group level, but they may also follow formalized agreements or wellestablished traditions for collaboration (D'Este et al. 2018). In all cases, the purposes and aims of the interacting organizations should be considered if the relations are to be evaluated and possibly improved. Spaapen and van Drooge (2011) 'understand productive interactions as exchanges between researchers and stakeholders in which knowledge is produced and valued that is both scientifically robust and socially relevant'. The results of these productive interactions may contribute to societal impact of research. While inspired by the concept of productive interactions, our definition of normal societal impact extends to also include results of interactions that are independent of specific scientific results. There may also be incidental or systematic interactions with societal partners where the expected results of the interactions are missing, impaired, or inadequate, or where the outcomes are neither scientifically robust nor socially relevant. Such shortcomings should be evaluated in relation to organizational purposes and aims. Normal societal impact is not unintended impact, as it is related to dedicated activities following strategic choices and objectives.
In contrast to normal impact, we define extraordinary societal impact as more rare incidences where traditional and typical or new and untypical interactions between science and society have unexpected widespread positive or negative implications for society. In this definition, we include extraordinary cases of negative impact ('grimpact', Derrick et al. 2018) as well. The Macchiarini case mentioned above is an example. But, of course, there is also a rich history of positive examples of ground-breaking results with unexpected wide-reaching societal impact.
Current methodology for evaluating the societal impact of research, evidence-based case studies, tends to select incidents of particularly interesting or impressive impact that can be traced back to the work of individual scientists. These incidents may be extraordinary also in the sense that they have unusually wide implications or demonstrate impact in new relations where impact normally does not occur, e.g. in the relation between the humanities and engineering for sustainable urban development. Contrary to such extraordinary impact-which by definition is rare, and often based upon serendipity-evaluation of normal impact implies a focus on the quality of everyday normal interactions between research and society in areas of research and sectors of society where such interaction can be expected from the purposes of the interacting organizations. This type of evaluation will take the perspective of both sides of the interaction.
In the following, we will first discuss some recently proposed or implemented frameworks for the understanding and evaluation of societal impact, using our main distinctions between normal and extraordinary impact and between the individual and organizational level of scientific work and science-society interactions. We will then use our distinctions to point out a new direction for research and development in this area before we conclude with a consideration of the policy implications of our proposals.

The REF methodology for evaluating societal impact
REF2014, the Research Excellence Framework for the evaluation and funding of universities in the UK, was the first broad ex post assessment of societal impact of research to be carried out (Derrick and Samuel 2017). It is also the most studied and discussed so far in the literature (Pedersen et al. 2019). The REF methodology has already been applied in several other countries, e.g. at the institutional level by the Royal Institute of Technology (KTH) in Sweden in 2014 and at the national level by the research evaluations performed by the Research Council of Norway since 2016.
The REF methodology requires evidence of societal impact related to specified and documented achievements in research. There is a template for the written case reports (REF2014 2012) which among other things demands the identification and documentation of: • The research that underpinned the impact: 'This section should outline the key research insights or findings that underpinned the impact, and provide details of what research was undertaken, when, and by whom'.
• The resulting impact: 'A clear explanation of the process or means through which the research led to, underpinned or made a contribution to the impact (for example how it was disseminated, how it came to influence users or beneficiaries, or how it came to be exploited, taken up or applied)'.
The typical analysis of case studies based on the REF methodology has been to identify pathways, beneficiaries, and effects of research in the reported cases, with a clear stance on excellence, not only in science but also in societal impact. In our view, this model for collecting and evaluating reported cases of societal impact is implicitly based on an understanding of societal impact that reminds us of the so-called linear model of innovation (Godin 2006) or communication (Shannon and Weaver 1949). It thereby has a basic problem with being at odds with most empirical studies of the science-society interactions in our time and what more theoretically has been called Mode 2 in the interactive dynamics between science and contemporary societies (Gibbons et al. 1994).
Moreover, the requirements in the REF to link scientific contributions, most often in publications, to demonstrable traces of societal impact is particularly exposed to some general problems with linking research activities to societal impacts. These are the problems with e.g.: • Causality: The relationships between research and innovation inputs, activities, outputs, and impacts are often unclear or nonlinear. • Attribution: It is difficult or even impossible to separate the impact of research and innovation from other inputs and activities. The REF is in the end about institutional funding. Inevitably, the REF methodology for evaluating societal impact is mostly focused on one side of the interaction. The case studies methodology also makes the universities report primarily examples of extraordinary impact, mostly at the individual level. This procedure has many valuable outcomes. It increases awareness of the societal responsibilities among researchers and provides strong stories to tell in the media. However, asking for the accountability of the research performing side of the interaction, it does not take its point of departure in societal needs. The evaluation of normal impact will instead approach the organizational level at both sides and ask-in specific and typical relations-how the interaction is functioning on a daily basis on both sides, according to organizational purposes and aims. This approach could provide evaluations to learn from. As a bonus, the four problems mentioned above (causality, attribution, internationality, and time scale) will become less important for the analysis. Other evidence about daily operations and their management and infrastructure will be in focus. Hence, the normal societal impact evaluation will reflect the formal, informal, and bidirectional

Alternative frameworks and methods
Potentially more in line with what we mentioned above as the Mode 2 theory of the interactive dynamics between science and contemporary societies are several other frameworks and projects for the understanding of the societal impact of research, such as the Payback Framework (Levitt et al., 2010;Klautzer et al. 2011), the SIAMPI/ERiC model (Spaapen et al. 2007 (Joly et al. 2015), and the Quality and Relevance in the Humanities QIRH (https://www.qrih.nl/en) project. An overview of such frameworks and projects is forthcoming from the Humanomics Research Programme at Aalborg University Copenhagen (Pedersen et al. 2019). Another overview, mostly focused on health research and outcomes, is found in Greenhalgh et al. (2016).
In these other frameworks for understanding and evaluation, the production and use of knowledge is understood as a process of interaction and co-creation rather than as a linear process that eventually leads to an effect or 'impact' outside of research. Many of these frameworks and projects collected empirical evidence. The first empirical data were collected by the Arthritis campaign and the UK Heart Foundation based on the Payback Framework. A number of cases exist on the basis of ERiC in the (H2020) SIAMPI project. ASIRPA also included a cross-sectional analysis of case studies. The empirical cases contribute to learning and understanding and emphasize that normal societal impact should be assessed at the organizational (or group) level. The other frameworks are valuable for their theoretical, conceptual, and empirical contributions to the field of research on societal impact, taking on board different philosophical assumptions ).
In the current situation with many valuable contributions to understanding and evaluating what is most often called societal impact, we think it is time to ask whether 'impact' is the right term at all. Does it lead to only asking for evidence of individual-level impact, focusing on only one side of the interaction, and requiring an extra effort of the employees of the research organizations to demonstrate value for money for authorities and/or funders? The quest for evidence of impact seems to assume that the science-society interaction is not normal but might sometimes take place in any unexpected place and only in particular and extraordinary cases. This assumption results in a burden of evidence on the researchers' side of the societal impact evaluation methodology. We think this burden could be relieved by replacing the term 'societal impact' with 'societal interaction' and a focus on the real and normal organizationallevel interaction according to the aims and purposes on both sides. Two questions could be asked to both sides: what are you doingdemonstrably-as an organization to take care of creating, exchanging, and making use of new knowledge according to your purposes? And what can we learn from this to improve-together? By taking both sides into account, sharing responsibility for science-society interactions becomes normal as well.

Normal interactions with society are different and typical for each field of research
The missions of general universities toward society are usually expressed in very general terms. Less vaguely expressed are the aims and purposes of research organizations with a more specialized profile (e.g. agricultural universities or public health research institutes).
Evaluations of normal impact will need this kind of specificity in their approach. It will be necessary to accept that societal relations differ by fields and subfields of research. This was clearly demonstrated by two recent national evaluations of the humanities (Research Council of Norway 2017) and social sciences (Research Council of Norway 2018). Both included evidence-based case studies and evaluations of societal impact according to the REF methodology. A few of the cases from the humanities demonstrated extraordinary contributions to information technology, bioethics, peace processes, emergency communication, and genetic counseling. The commissioner of the evaluations, the Research Council of Norway, chose to highlight these extraordinary cases when reporting from the exercise. However, after looking through all cases ourselves, we could clearly see that the social sciences and humanities (SSH) more typically and normally contribute to other areas of society: social welfare, policy design, public administration, international affairs, integration and understanding of different languages and cultures, education at all levels, cultural life, media and information, and history, the 'memory of society'. The case studies also demonstrated that research in the SSH is integrated in, and not operating at a distance from, certain domains in society where the disciplines may have specific purposes and play specific roles in specific societal and cultural contexts. Musicology usually contributes to musical life and research in international relations normally to diplomacy and foreign policy.
These purposes and roles may often be more specific than seen in a general typology or description of pathways, beneficiaries, and effects. Examples of such generalizations may be 'improving health and well-being' or 'commercialization and exploitation'. At the same time, the specific aims of the research-society interaction may be more general than the individual case report can account for. Hence, a more specific typology of typical societal relations in each field of research is needed. We will give an example.
Law studies are concentrated in the universities' faculty of law in most countries. The typical interaction with society of a faculty of law is different from other faculties and at the same time more specific than the general societal responsibility of its university: A faculty of law serves the legal system of a country by educating professionals and responding to societal needs in the legal system. This service is much more specific than 'enhancing the effectiveness and sustainability of organizations including public services and businesses' (Pathways to impact, Research Councils UK). It also needs to be specified within the faculty. Studies in e.g. EU Law (the research is international in focus and applications) or Criminal Law (the research is national in focus and directly concerned with the civil society) will have different relations to society. Such specific relations need to be understood before they are evaluated. Extraordinary cases of particularly impressing impact will not be sufficient for such an understanding. They cannot serve a real evaluation to learn from; and they will only provide an 'exhibit' for publicity.

Involving stakeholders and improving relations
All frameworks for understanding and evaluating impact of research mentioned above, including the REF, have in common that their major focus at the end is on evaluating the research performing side of the interaction with society. This one-sided focus is understandable since the frameworks have been developed for research funding organizations and their needs. However, if the purpose of an evaluation is to be formative more than summative (not only assuring value for money but improve by learning from experience and advice), and societal impact is studied as an interaction, both sides of the interaction should be able to learn from the evaluation. Stakeholder engagement is not only to inform them on the research, but also to include their views and needs in the research practice (D'Este 2018). Similarly, attempts to evaluate the interactions between frequent and typical interactors at the organizational level should strive to involve and make the evaluation useful for both sides.
A societal organization might even be very interested in an evaluation of its ability to interact with, influence, and continuously learn from new research. The Dutch Heart Foundation is a good example of a charity funding organization that aims to give influence and input to their societal partners. Recognizing that research funding is their core business, they have developed strategies and practices to have two-sided interactions between research and potential users of research in the early phases of research, namely in (1) research agenda setting and (b) evaluation of research proposals. Regarding the former, professionals and a group of no less than 11,000 citizens helped in prioritizing the research agenda, limiting it to five key topics. Regarding the latter, an end users committee evaluates research proposals in parallel to the scientific advisory board, using criteria of relevance, participation and activities, and interactions with users. This committee consists of professionals, (heart) patients, and citizens. All of these science-society interactions concern normal practices that are steered with a responsibility at the organizational level.

Policy implications
We have used the distinctions between normal and extraordinary societal impact and between organizational-and individual-level activities and responsibilities to discuss how science-society relations can better be understood, evaluated, and improved by focusing on the organizations that typically interact in a specific domain of research. We will conclude by pointing at policy implications.
Our perspective on societal impact evaluation is in line with the official policy for Responsible Research and Innovation (RRI) in the Horizon 2020 programme of the European Union (Stilgoe et al. 2013;von Schomberg 2013). With the perhaps clearest statement so far of the expected societal relevance of research, the policy 'implies that societal actors (researchers, citizens, policy makers, business, third sector organizations, etc.) work together during the whole research and innovation process in order to better align both the process and its outcomes with the values, needs, and expectations of society'. RRI essentially is sharing responsibility and depends on groups and organizations rather than on individuals. Our advice is to perform societal impact evaluation accordingly. We suggest to: Focus on normal impact rather than extraordinary impact: Societal impact of research is normal and part of society. Normal impact is about daily activities and how well they are organized, not about individual incidents of particularly interesting or impressive impact.
Focus on relations and interactions: Societal impact evaluation needs to consider both sides in the relations between research and society. The main purpose of the evaluation should be the improvement of the relations, rather than the assessment or funding of one side of the relation. The present typology of impact, often called pathways to impact (e.g. cultural and heritage preservation) needs to be supplemented by an identification of the relevant interactors or sectors in society, resulting in a typology of interacting organizations (e.g. museums).
Understand the diversity of purposes of interactions: Fields of research have different relevance for society, and the organizations on both sides have specific purposes for interacting.
Apply an organizational-level perspective: In general, normal societal impact with possible positive effects can be seen as an organizational-level responsibility, not just as the responsibility of each individual researcher. An organizational-level evaluation may focus on how well the systematic interaction is taken care of in the strategies, infrastructures, management, incentives and rewards, and daily life of the organizations. The organizational-level perspective may also better serve the implementation and follow-up of a formative societal impact evaluation.

Funding
This work was supported by the Research Council of Norway, grant number 256223.