Abstract

This empirical study explored how research can generate impacts by investigating different sorts of impacts from one academic field—mathematics—and the diverse mechanisms generating them. The multi-method study triangulated across: (1 and 2) content analysis of impact case studies and environment descriptions submitted to the UK Research Excellence Framework (REF) assessment; (3 and 4) a survey and focus group of heads of mathematics departments; and (5) semi-structured interviews. Mathematics has had a full range of impact types, particularly conceptual impacts, although more tangible instrumental impacts were prioritized for REF. Multiple mechanisms were utilized, but seldom appeared in REF case studies. Long-term relationship building and interdisciplinarity are particularly important. Departmental culture and certain knowledge intermediaries can play proactive roles. In sharp contrast to simplistic linear narratives, we suggest that appreciation of diverse impact types, multiple, often informal, mechanisms and dynamic environments will enhance the likelihood of meaningful impacts being generated.

1. Introduction

As countries around the world increasingly demand ‘return’ on investment in research, universities strive to demonstrate their influence, often through government-related assessment of impacts. Along with this has been a growth in academic literature exploring and documenting processes of knowledge exchange, and ways in which research-derived knowledge can generate impacts. We conducted an in-depth, multi-method study of how research in one academic field—mathematics—has given rise to a number of types of impacts.

Eminent Cambridge mathematician G. H. Hardy remarked:

‘It is not possible to justify the life of any genuine professional mathematician on the ground of the “utility” of his work.’ (Hardy 1940)

Hardy is known for his work on the mathematics of prime numbers, which, ironically, underpins modern cryptography and the security of every transaction on the internet, demonstrating enduring utility of mathematics in practical science and everyday life. A report for the UK government (Deloitte 2013) estimated the contribution of mathematical science to the economy in 2010 to be 2.8 million in employment terms (around 10% of all UK jobs) and £208 billion in terms of gross value added (around 16% of total UK). (Of course, any such attributions of economic ‘impacts’ of often fundamental knowledge are of necessity dependent upon assumptions of the particular model used.)

Mathematics therefore presents a triple conundrum to those studying impacts and their generation:

  • mathematics, and activities of a mathematical nature in fields such as computing, communications, engineering, and finance, permeate every aspect of modern life;

  • many of its elite practitioners present mathematics as a cultural phenomenon, and argue vehemently that a focus on impact is a threat to the health of the discipline; and

  • compared to the physical sciences it scores very low on many traditional metrics of knowledge exchange, for example patents, licences, and spin-outs.

Tackling this challenging triple conundrum, posed by a field often seen as quintessentially academic, yet nonetheless giving rise to multiple impacts, may help to illuminate impact generation more generally. Here, we view mathematics through an approach originally developed for social sciences, which augments the more usual instrumental impacts e.g. through exploitation of intellectual property (IP) with those that develop new concepts and create new capacity. We work from a conceptual framework for impact originally developed by Nutley, Walter and Davies (2007), refined by Meagher, Lyall and Nutley (2008) and Meagher and Lyall (2013), which highlights a rich ecosystem of varied actors, roles, and flows of knowledge, expertise and influence between them, and draws attention to the diversity of mechanisms in use by individuals and institutions to create and sustain impact. At the heart are five kinds of impact:

  • conceptual impacts: generating new understanding or raising awareness among potential users of research findings;

  • instrumental impacts: ‘tangible’ products or services taken up by companies, policymakers, or practitioners;

  • capacity building impacts: training and/or developing collaborative abilities;

  • attitude or cultural change: ‘increased willingness’ to engage in knowledge exchange activity, on the part of individuals, institutions, or organizations; and

  • enduring connectivity: establishment of long-lived external relationships.

1.1 Mathematics, mathematicians, and impact

Steingart (2013) describes the transformation of mathematics in the USA during the mid-twentieth century, in which the traditional humanistic approach of those, who, like G. H. Hardy, saw mathematics as an intellectual endeavour pursued for its own sake, was overtaken by an instrumental view of the discipline as contributing to the nation. Vannevar Bush (1945) influentially articulated a long-term linear model, whereby unfettered intellectual curiosity of scientists gives rise to unpredictable and useful discoveries, and this argument endures. For example, the American Mathematical Society testified to government:

‘Society has benefited from the many products, procedures, and methods resulting from NSF supported [mathematics] research—research performed over many years and not always predetermined toward specific applications. These benefits include well known innovations such as Google, magnetic resonance imaging (MRI), and bar code technology.’ (Vogan 2013)

Similarly the UK’s Council for Mathematical Sciences argues for mathematics ‘creating high returns on investment for the UK’ (CMS 2015). There were, and are, dissident voices, for example distinguished UK mathematician Peter Cameron argues a focus on impact threatens to devalue curiosity-driven research, whose outcomes are unpredictable (Cameron 2011). However, a 2015 survey (Hughes et al. 2016) of UK academics’ attitudes suggested that among those in physics and mathematics, 39% were devoted primarily to basic research, 30% to applied research, and 29% to use-inspired basic research.

1.2 Capturing and incentivizing impacts

UK mathematicians work within the context of an impact agenda, which has affected both strands of government research funding: competitive research grants, and centralized formula funding known as ‘quality-related research funding’ (QR).

The UK Engineering and Physical Sciences Research Council (EPSRC) in the past provided focused grants for stimulating impact. For example, EPSRC funded a 2009–13 initiative at Queen Mary University of London, which provided pump-priming for: use of statistics in law courts; optimizing databases for IBM; new models for emergency room triage; and initial work towards mathematics-based start-ups, one sold to Facebook and another floated on AIM. However a recent EPSRC requirement to address impact in all research proposals brought particular hostility from the mathematical community.

Although some government funding supports university commercialization activity, advocacy groups caution against over-optimism (CMS 2015):

‘We argue that existing incentives, policies and funding streams for such relations, which may work well in other disciplines, are not well suited to mathematics’

noting the importance of factors such as strong ‘co-creation’, mutual enthusiasm, shared visions, and personal relationships.

Most QR funding is currently allocated by the ‘Research Excellence Framework’ (REF) (REF UK 2014): universities submit an evidence base for each discipline (‘Unit of assessment’ or UOA), and assessments, scaled by staff numbers, determine funding for each university. Controversially, the 2014 REF required universities to submit evidence of research impacts, to be assessed on ‘reach and significance of impacts on the economy, society and/or culture’.

Each UOA submitted ‘Impact Case Studies’ (hereafter ICS), whose format required underlying research papers and evidence of impact, together with a broader description of context and strategy (hereafter ‘Impact Environment Template’, IET. (A subsequent review of the role of metrics in the REF (Wilsdon 2015) captured multiple concerns over reliance on metrics, suggesting, for example, that, when assessing impact, replacement of narrative case studies with quantitative indicators is not feasible.)

Response from some sections of the UK mathematics community was hostile, both to the notion of assessing impact and the approach chosen. One learned society asserted baldly

‘the concept of Impact as formulated for the REF fails to recognise the key mechanisms through which the mathematical sciences achieve impact on science, industry, the economy and culture.’ (LMS 2011)

1.3 Broader evaluation of impact

Bodies funding or promoting research have attempted to demonstrate to national governments the value of ‘return on investment’ in research (RCUK 2007; Nature Outlook 2014; NRC 2014). Other organizations have attempted to capture and promote impacts of ‘their’ fields; organizations oriented towards innovation, economic development, or business have also identified research contributions.

While attempts to evaluate impacts of research have become more frequent and widespread, they have not become simple. Despite serving as a critical resource on analysis of the relationship(s) between research and its role beyond academia, Nutley, Walter and Davies (2007: 283–284) nonetheless illustrate the complexity and diffuseness of impacts.

Emphasis is often placed on challenges of capturing impacts, such as ‘proving’ causality or attribution (RCUK 2007; Grant et al. 2010), the frequent time lag that occurs between research and often indirect impact (Molas-Gallart, Tang and Morrow 2000; Roux et al. 2010), or even achieving a common understanding of what ‘impact’ means. Samuel and Derrick (2015) interviewed 62 evaluators prior to their assessment roles for the REF health-related panel and its sub-panels, finding significant heterogeneity in their expectations of how to characterize societal impact.

An extensive literature has grown on knowledge exchange (KE) (Nutley, Walter and Davies 2007). Views of the process have undergone significant change over time—from a one-way research ‘push’ model of ‘technology transfer’, through to increasing emphasis on two-way dialogue between researcher and stakeholder (‘knowledge exchange’), and now frequently incorporating strong roles for stakeholders in ‘co-production’ (Armstrong and Alsop 2010) and emphasizing how multiple types of knowledge move towards having an influence in different contexts, as in ‘knowledge mobilisation’ (Davies, Powell and Nutley 2016). A range of thoughtful papers (Davies, Nutley and Walter 2005; Bannister and Hardill 2013; Boaz, Locock and Ward 2015) have engaged with challenges of both defining diverse impacts and understanding processes which can lead to them. Key factors such as building trusting relationships, (Jagosh et al. 2015), indicators such as ‘rapport’ in ‘mature partnerships’ (Kothari et al. 2011), and roles such as that of ‘knowledge intermediary’ (Lightowler and Knight 2013) have been identified. Analyses have often focused on relatively ‘applied’ areas such as health (Lomas 2000; Davies, Powell and Nutley 2016), the environment (Phillipson et al. 2012; Fazey et al. 2014; Reed et al. 2014), and education (Cooper 2014) to contribute to more general learning. Despite significant increase in the knowledge mobilization literature, Powell, Davies and Nutley (2016) draw upon an empirical study of research agencies to articulate an ironic challenge: theory and practice are not articulated fully regarding knowledge mobilization itself.

We underscore that, ideally, analysis of impacts and elucidation of practical impact-generating mechanisms can be closely intertwined to the benefit of both aims. This perspective made it possible, for example, for Meagher and Lyall (2013) to draw upon impact evaluations of several quite different research programmes to shed light on key mechanisms including that of knowledge intermediary. Emphasizing the importance of interactions between research and various stakeholders in society, Spaapen and van Drooge (2011) suggest that an appropriate framework for impact evaluation can act as a tool for ‘enlightenment’, enabling researchers and stakeholders to understand steps that can lead towards social impact; they suggest focusing on ‘productive interactions’ of three kinds: direct/personal interactions, indirect (e.g. through texts), and financial interactions (e.g. through contracts). Molas-Gallart and Tang (2011) employed this approach to analyse a social sciences centre and found its emphasis on productive interactions useful in understanding processes leading to social impacts. Of particular relevance to our questions is the close analysis of REF community-based health sciences ICS (Greenhalgh and Fahy 2015), in which the authors found (as did we) little express attention paid to impact-generating mechanisms. They identified a ‘mismatch’ between the ‘direct’ and ‘linear’ depictions of impact emergence submitted to the REF, and what is recognized in the literature as the importance of indirect, long-term (and less readily attributable) impacts, and their collaborative generation.

Our hypothesis is that teasing out types of impacts and the diversity of mechanisms contributing to their generation within the academic field of mathematics will shed further light on the richly textured mechanisms through which impacts are generated. We framed our approach with the set of five types of impacts used in other studies, described above, not only to test what we expected to be their utility in capturing types of impacts from mathematics research, but also to utilize them in ‘opening up’ a picture of what we predicted would be a range of mechanisms involved in impact generation. By using this framework for our study, we hope to refine understanding and enhance its applicability for others analysing the evaluation and generation of impacts, or indeed taking action to bring about impacts.

2. Methods

2.1 Multi-method study

Our study was grounded in a conceptual model considering research impact to be a function of the interaction between the content of the research, the context for its application, and the processes of user engagement, with those processes including multidirectional flows of knowledge, expertise, and influence across a web of networks and relationships (Meagher, Lyall and Nutley 2008). For this study, we devised a common framework of core questions to be pursued, centring upon impacts and impact-generating mechanisms or processes; each was addressed by multiple methods, thus facilitating rich analysis triangulating across methods to draw together findings. The methods were:

  • close content analysis of 209 REF ICS submitted by 51 mathematics units (excluding redacted case studies);

  • close content analysis of 52 REF ‘Impact Environment Templates’ submitted by mathematics units;

  • survey of heads of mathematics departments;

  • focus group with heads of mathematics departments; and

  • semi-structured interviews with 23 individuals having overview perspectives.

Importantly, this study was not an analysis of the REF itself; while two of the five strands analysed REF submissions, we did not consider assessment outcomes. Whereas the REF prioritized relatively tangible impacts, we prioritized references to the nature of impacts and mechanisms of impact generation.

While institutions were selective in their choice and presentation of ICSs, so as to maximize assessment outcomes, nonetheless the REF documents represent a comprehensive set of comparably derived narratives of impact amenable to standardized interrogation. To ensure more rounded understanding, the two-pronged document analysis was complemented by a set of qualitative methods, including a survey, interviews, and a focus group. Heads of departments were selected as a key source due to understanding of the field, how it generates impacts, and submission tactics. Naturally, methods were complementary to each other. For example, in addition to semi-structured interviews gaining insights from stakeholders and others with ‘overview perspectives’, semi-structured interviews of heads of departments made it possible to probe attitudes towards impacts, processes, and tactics more deeply than did their surveys, while the surveys gathered information from a greater number of heads of departments. We targeted the focus group towards conceptual impacts to engage heads of departments with one particular type of impact, and relevant mechanisms, at some depth. However, at the same time, complementarity did not equate to assignment of one method to one question: questions were not addressed exclusively by just one method. To make the most of the opportunity for enriched understanding regarding each question, we took care to ensure that the same concepts underpinned all the methods so that findings could be integrated. Thus, for example, a set of five types of possible impacts were defined consistently in the same way for all methods—even though the opportunity for discussion during interviews made it possible to actually explore the subtle, process-based impact types in more depth than was possible in surveys. Our working definition for any one mechanism remained the same across methods, although some methods (e.g. analysis of IETs) provided more numerous mechanisms than others, in some sense a counterweight to the inevitable risk of subjectivity in identifying mechanisms. We view the multiple methods as ‘strands’ which, when woven together, contribute in a robust way to understanding.

2.2 Content analysis, ICS, and IETs

We carried out a close content analysis of all 209 non-redacted REF ICSs for the Mathematical Sciences Unit of Assessment (REF UK 2014) to look for reports of different kinds of impact, following (Meagher and Lyall 2013), and of mechanisms used to generate those impacts. Similarly, we analysed all 52 IETs to identify mechanisms reported. Throughout this analysis, language such as ‘mechanism X was used’ is shorthand for ‘mechanism X was described as having been used’—we could not assess veracity. Any scoring based on textual analysis has an inevitable element of subjectivity which we minimized by focussing carefully on points that were actually articulated, avoiding inference, although inevitably dependent on authors’ wording.

For ICSs, a pilot analysis of a subset of units allowed us to clarify how we assigned descriptions of impacts to our five initial categories. Likewise, the pilot allowed identification of mechanisms to emerge; we generated a final list of 12 impact-generating mechanisms (Table 1, Column 2). For example, we observed instances of substantial work with a company where some formal mechanism must have existed, and we defined a new mechanism ‘Formal, not specified’; likewise, the role of free software as a vector for new ideas became apparent. We each read closely all 209 ICSs, and conducted (separately) analysis identifying any impact or mechanism cited explicitly. These were compared and any recurring differences discussed, allowing us to align our results, although differences were few. The resultant 209 case studies and 12 mechanisms gave 2,508 data points for further analysis.

Table 1.

Mechanisms from across data sources

Heads of Department
 
ICS
 
Impact templates
 
Mechanism (%) Mechanism (%) Mechanism (%) 
Interdisciplinarity 82 Interdisciplinarity 35 Interdisciplinarity 83 
Interdisciplinary uptake 36     
Informal relationships 68 Informal KE Relationships from other relationships 63 
    Long-term relationships, general 50 
    Long-term relationships with named individuals 35 
Joint publications 50     
Co-production 46 Co-production 54   
Visits—outgoing 46   Secondments—outgoing 44 
Visits—incoming 23   Visits incoming 17 
    Other visits 23 
Public engagement 41 Public engagement 19 Public engagement 71 
    Media 29 
PhD student or post-doc 36 Movement of PhD 19   
Tailored events 36 Presentations, events, etc. 45 Sustained events 58 
Tailored presentations to stakeholder events 14     
CPD 23   Capacity-building activity in stakeholders 31 
Knowledge Intermediary Knowledge Intermediary including industry consortia, KTPs, etc. 37 Knowledge Intermediary 56 
Consortium including industry 59     
KT Network/partnership 55     
  Free software 16 Ready access to research findings and software, e.g. on websites 24 
    Alumni 23 
Contract research 73 Contract research 24 Stakeholder funding 46 
Consulting 68 Consulting 11 Consulting 54 
Patent, licence, or spin-out Patent, licence, or spin-out 14   
  Formal mechanism, not specified 33   
Heads of Department
 
ICS
 
Impact templates
 
Mechanism (%) Mechanism (%) Mechanism (%) 
Interdisciplinarity 82 Interdisciplinarity 35 Interdisciplinarity 83 
Interdisciplinary uptake 36     
Informal relationships 68 Informal KE Relationships from other relationships 63 
    Long-term relationships, general 50 
    Long-term relationships with named individuals 35 
Joint publications 50     
Co-production 46 Co-production 54   
Visits—outgoing 46   Secondments—outgoing 44 
Visits—incoming 23   Visits incoming 17 
    Other visits 23 
Public engagement 41 Public engagement 19 Public engagement 71 
    Media 29 
PhD student or post-doc 36 Movement of PhD 19   
Tailored events 36 Presentations, events, etc. 45 Sustained events 58 
Tailored presentations to stakeholder events 14     
CPD 23   Capacity-building activity in stakeholders 31 
Knowledge Intermediary Knowledge Intermediary including industry consortia, KTPs, etc. 37 Knowledge Intermediary 56 
Consortium including industry 59     
KT Network/partnership 55     
  Free software 16 Ready access to research findings and software, e.g. on websites 24 
    Alumni 23 
Contract research 73 Contract research 24 Stakeholder funding 46 
Consulting 68 Consulting 11 Consulting 54 
Patent, licence, or spin-out Patent, licence, or spin-out 14   
  Formal mechanism, not specified 33   

For the IETs, our premise was that, as encouraged by looser regulations, authors might have felt more free to discuss intangibles and processes. To broaden and deepen our understanding, we developed a list of 27 mechanisms (Table 1, Column 3; Table 2), again fine-tuned in light of a pilot analysis, for example differentiating between appointments/recruitment and promotion/appraisal. Only activities that had occurred were considered, not future plans, and statements were taken at face value. The final data, 27 mechanisms across 52 units (1,404 data points), offer a rich view of impact-related activity across UK mathematics.

Table 2.

Management mechanisms

Management of the environment mechanisms reported in IETs Percentage observed 
Management of the environment  
University structure and resources 81 
Early career researcher training 69 
Departmental structure—key responsibilities 60 
Time/workload allocation 54 
Promotion/appraisal 50 
Department funding 44 
Appointments and recruitment 42 
Reward (recognition, e.g. through leave, of achievements in knowledge exchange) 38 
Presence in unit’s strategic plan document 31 
Training and preparation of academics in knowledge exchange 29 
Culture (explicit commitment to creating a culture conducive to impacts) 19 
Management of the environment mechanisms reported in IETs Percentage observed 
Management of the environment  
University structure and resources 81 
Early career researcher training 69 
Departmental structure—key responsibilities 60 
Time/workload allocation 54 
Promotion/appraisal 50 
Department funding 44 
Appointments and recruitment 42 
Reward (recognition, e.g. through leave, of achievements in knowledge exchange) 38 
Presence in unit’s strategic plan document 31 
Training and preparation of academics in knowledge exchange 29 
Culture (explicit commitment to creating a culture conducive to impacts) 19 

2.3 Survey—heads of mathematics departments

The chair of the UK Heads of Departments of Mathematics (HoDoMs) emailed invitations to our online (SurveyMonkey) survey addressing 152 recipients, about 75 of whom are heads of mathematics units. Twenty-nine responses were received, representing 24 different institutions, nearly a third of the institutions belonging to HoDoMS. Introduced as ‘exploring the various ways in which mathematicians generate a range of impacts over time, not all of which may be obvious’, the survey collected a mix of Likert scale, precoded responses, and free-text responses. (Percentages given in survey results refer to percentage of responses to that question.) Drawing on extensive experience with online surveys relating to research impacts, we composed and reviewed carefully the 20-question survey, half (11) of which were Likert scale, three of which were precoded lists, and six (including ‘Other Comments’) were free text. Near the beginning of the survey, definitions of the five impact types were framed carefully (leading to a Likert question and a precoded question), as they have been in multiple other surveys; these same definitions were used to introduce discussion of impact types during the semi-structured interviews. The principal precoded list consisted of 16 different mechanisms/activities (as well as ‘Other’); this list was devised through dialogue between the two co-authors, both with extensive experience in this area. No precoded list can be perfect, but an opportunity to add in ‘Other’ mechanisms/activities was provided, and the range of choices was double-checked for ready comprehension and reasonably diverse coverage, given usual balances with survey brevity. (A reader interested in such methods might find (Bryman 2012) useful.)

2.4 Semi-structured interviews

A qualitative analysis of interviews based on a semi-structured interview template provided triangulation across views of 23 informed individuals. While some overlap existed (e.g. department heads sometimes play additional roles), the distribution was, roughly, 11 individuals with ‘overview or big picture perspectives’ (e.g. holding a learned society office, serving on the REF panel), four non-academic stakeholders, and eight heads of mathematics/mathematics-related departments. This number of interviewees could not purport to be representative of all individuals involved in mathematics; rather, for purposes of in-depth pursuit of the study’s questions, they were selected as individuals known by the mathematician co-author/colleagues to be thoughtful and willing to reflect on complex issues, while coming from the range of perspectives described. Conducted by telephone, interviews were on average 45–60 minutes long.

2.5 Focus group, heads of mathematics departments

At the April 2015 annual conference of HoDoMs, we led a 2-hour focus group on conceptual impacts and impact-generating mechanisms, attended by 36 heads of department or their representatives, and 4 other senior opinion formers. The study and definitions of the five types of impacts were introduced by the mathematician co-author, and the focus group targeting conceptual impacts was facilitated by the other co-author, an experienced facilitator. With lively discussion, small subgroups of two to three wrote on cards provided: (1) examples of conceptual impacts from mathematics generally; (2) effective mechanisms or steps to making conceptual impacts happen; and (3) ways in which awareness and development of conceptual impacts might be enhanced in the mathematical community. This input was typed and analysed.

3. Evidence and findings

3.1 Impacts seen in the ICS

We found that mathematics research had a great deal of impact in various impact categories. (Others have identified through REF the diversity of sectors reached: EPSRC found linkages to all 22 industry sectors (EPSRC 2015); and a PESTLE analysis within a text mining analysis across all fields highlighted the distinctive breadth of influence of mathematics on non-technological areas (King's College London and Digital Science 2015).

Across 209 ICSs, we found a preponderance of instrumental impacts due to wide expectation that this is what the REF valued. (This was not unfounded, given the guidance provided for units in natural sciences: for some other fields the language seemed more encompassing.) Of the other categories, there were many mentions of conceptual impacts (although relatively few ICSs revolved solely around them), with much less evidence of the remaining categories. All but 10 showed either Instrumental or conceptual impact, with 106 showing both (Tables 3 and 4).

Table 3.

Frequency of impact types

Impact type Number of ICS where seen Percentage of ICS where seen 
Instrumental 179 86 
Conceptual 126 60 
Capacity-building 42 20 
Enduring connectivity 39 18 
Attitude/culture change 
Impact type Number of ICS where seen Percentage of ICS where seen 
Instrumental 179 86 
Conceptual 126 60 
Capacity-building 42 20 
Enduring connectivity 39 18 
Attitude/culture change 
Table 4.

Conceptual and instrumental impacts

 Instrumental, total 179 Not instrumental, total 30 
Conceptual, total 126 106 20 
Not conceptual, total 83 73 10 
 Instrumental, total 179 Not instrumental, total 30 
Conceptual, total 126 106 20 
Not conceptual, total 83 73 10 

3.2 Impacts known versus those reported to REF

Anonymous surveys showed a sharp contrast between the variety of impacts known by department heads, and the narrow distribution of types of impacts they reported having actually submitted to the REF (Table 5). All but one (96%) were aware of Instrumental Impacts arising from their departments’ research; yet three quarters were also aware of each of conceptual, capacity building, and enduring connectivity; and half saw attitudinal or cultural impacts (it is possible that some respondents included public outreach in this). By far, instrumental impacts were noted by the highest percentage of respondents to both questions, fitting with REF ‘rules of the game’. In contrast to the 90% of respondents who had submitted Instrumental Impacts, submission percentages for other types of impacts fell sharply in comparison to percentages of those known, suggesting a tactical decision towards submitting Instrumental Impacts. Interestingly, Conceptual impacts—while submitted far less frequently than seen—nonetheless were submitted by just over half of the respondents, suggesting that respondents viewed them as important, even if (as indicated in interviews) they might not have been as confident of their REF valuation.

Table 5.

Types of impact existing, submitted, and seen

Impact type % HoDs aware of this in their department % HoDs reporting submitting in an ICS % of ICS where this impact seen % UOAs where this impact seen in at least one ICS 
Instrumental 96 91 86 94 
Conceptual 75 55 60 92 
Capacity-building 79 27 20 51 
Attitude/culture change 50 22 
Enduring connectivity 75 13 18 43 
Impact type % HoDs aware of this in their department % HoDs reporting submitting in an ICS % of ICS where this impact seen % UOAs where this impact seen in at least one ICS 
Instrumental 96 91 86 94 
Conceptual 75 55 60 92 
Capacity-building 79 27 20 51 
Attitude/culture change 50 22 
Enduring connectivity 75 13 18 43 

Respondents’ percentages of REF submission (91% instrumental impacts; 55% conceptual impact) were well-aligned with percentages we found in our analysis of ICSs (86, 60%, respectively). Conventional university-based capacity-building impacts, clearly observed by these academic leaders, were disallowed by the rules and seldom submitted. Not surprisingly, given REF requirements, the two ‘process-based’ impacts—enduring connectivity and attitude/culture change—were very seldom submitted, despite three quarters and half of the respondents, respectively, being aware of them. Like the respondents’ views of submissions, our case study analysis also reflected a very sharp drop in percentages of these last three types of impact. In Table 4, the differences between the third column (% of HoDs submitting) and last column (% of UOAs) may be due to only half of department heads responding, likely thinking of only main ICS elements, whereas our ICS analysis documented all impacts. Cumulatively, across a UOA’s total submission (including multiple ICSs), percentages for all types of impact other than attitude/culture change were higher, suggesting that, in effect if not by design, institutions were submitting sets of somewhat diverse case study stories/impacts.

3.3 Interview evidence on impacts

Interviewees saw the broader notions of impact as more relevant to mathematics, but harder to evidence with the concrete path of causality that the REF required. Running throughout our conversations, when the REF was still fresh in interviewees’ minds, was a tension between what was seen by interviewees as important versus a narrow view of impact promoted by the REF, with its apparent stress on instrumental impacts. (Indeed, whatever the ‘official’ intent of the REF, interviews with heads of departments described pressures, e.g. from their college or university administrations, to behave in highly tactical ways in selecting and writing impact case studies.) Many interviewees felt that the emphasis on instrumental impacts in ICSs did not play to mathematics’ strengths, diminishing the far-reaching and underpinning nature of its influence. As one HoD interviewee remarked:

‘This is a real blunt instrument; I hope impact can be measured in a more subtle way next time. The steer we got—we would be principally evaluated by instrumental impact measure; most maths departments were stuck on that. Most were forced to address “have we given some technology to some company that then made money with it?” Certainly for mathematics, instrumental impacts are the least frequent.’

For many interviewees, our stated definition of conceptual impact was particularly welcome, as it seemed to legitimize impacts that might otherwise be ignored or undervalued. The HoDoMs focus group also stressed that conceptual impacts are key to contributions of mathematics research, enthusiastically citing many examples. Overwhelmingly, and passionately, interviewees wanted to emphasize capacity-building impacts such as education of undergraduates and postgraduates who take mathematical understanding into ‘real-world’ environments, and production of textbooks and other educational material for use beyond universities, especially in schools. Some interviewees cited activities that build capacity in potential users, such as short courses or workshops (especially in statistics), and which might lay groundwork for future collaboration.

Such activity was seen as forming part of enduring connectivity impacts, discussed positively and with interest by perhaps half of both the overview and HoD interviewees. Comments, illustrated with examples, indicated the novelty of thinking of the achievement of ongoing interactions as itself a type of impact, while also recognizing the vital nature of long-term personal relationships (see Section 3.6).

The process-based impact of attitude/culture change was often difficult for interviewees to pin down, though it seemed that some, especially those with overview responsibilities, felt that some mathematicians’ attitudes and culture have changed towards greater engagement with non-academics.

3.4 Time to impact

A full 91% of respondents believed that ‘in general, a long timeframe elapses between mathematics research and the development of impacts’; two-thirds strongly agreed with this. Not surprisingly, most (86%) respondents felt that ‘it is important to me as a department head that my department retains an appropriate balance between long-term foundational research and research which is more immediately applicable’; two-thirds (68%) strongly agreed. The issue of timescale appears related to frequent interviewee comments on the importance of interdisciplinarity for the impact of mathematics, especially conceptual impacts.

3.5 Mechanisms of impact generation

3.5.1 Multiple strands of data

A central question of this study was how—through what mechanisms—were impacts generated? To answer this, we investigated several strands of data. ICS narratives were constrained by length and purpose to emphasizing impacts, such that mention of mechanisms was almost incidental, so numbers are probably unrealistically low. In contrast, IET narratives were meant to convince assessors that departments were doing all they could to produce impacts, allowing us to capture far more, and more fine-tuned, mechanisms, with the 27 mechanisms we found in the IETs more than double the 12 mechanisms in the ICSs. In between was the survey’s list of 18 mechanisms. Thus, there is not exact consistency across specific mechanisms in these three lists, although similar clusters of mechanisms emerge, with Table 1 allowing for a general ‘read-across’.

3.5.2 Frequency of use of mechanisms

Relative frequency of use of various mechanisms is illuminated in slightly different ways by each of the three strands (Table 1). The three strands are represented by three columns of data: (1) ‘Heads of Department’ provides percentages of mechanisms ticked by Heads of Departments in their 29 surveys; (2) ‘Impact case studies’ provides percentages of the 209 ICS in which the mechanisms were identified through close content analysis; and (3) ‘Impact Templates’ provides the percentage of the 52 units whose IETs mentioned particular mechanisms. Low numbers were found in ICSs compared to the other two data sources; only one mechanism (co-production) appeared in more than half (54%) of the ICSs.

In contrast, when HoDs were asked directly about a survey list of 18 possible mechanisms, four mechanisms were selected by over two-thirds: two as types of relationship building (interdisciplinarity by 82% and Informal relationships/knowledge exchange by 68%) and two as funding mechanisms. We highlight the high numbers citing the latter two, contract research and consulting, although these only appear as mechanisms in 24 and 11% of the ICSs, respectively, with perhaps some further occurrences subsumed under ‘Formal, not specified’. We return in the Discussion to the importance of interdisciplinarity and relationship building, and conversely, to the low percentage (under 10%) mentioning knowledge intermediaries and conventional IP exploitation mechanisms of patents, licensing, and spin-outs.

Corresponding analysis of the more nuanced 52 IETs, using 27 mechanisms, demonstrated that different mechanisms were utilized by different numbers of UoAs. A quarter (7, or 26%) of the 27 mechanisms assessed were used by 30 or more units. Including these, just under half (12, or 44%) of the mechanisms were used by at least half the units (26 or more). A quarter of the mechanisms (7, or 26%) were used only by 15 or fewer units. We note again the high proportion citing interdisciplinarity, along with various forms of relationship building. Frequently, these narratives incorporated explicit descriptions of one relationship building upon another as a route to impacts achieved. Mechanisms related to deliberate management (Table 2) appear only in this source.

3.6 Highlighted mechanisms

Four key mechanisms are not only important in their own right but also, we suggest, point to important challenges in considering impact generation and evaluation.

Interdisciplinarity—This emerges repeatedly as a key mechanism towards impacts for mathematics.

Relationshipbuilding—This is manifestly important, judging by nearly all sources (and, indeed, most literature on knowledge exchange), but is nearly invisible in the REF ICSs.

Knowledge intermediaries and exploitation of IP—While increasingly recognized as important, generally, views among mathematicians point to caveats.

Culture—While often subtle, insights into culture could lead quite directly to practical considerations in generating impact.

3.6.1 Interdisciplinarity

The mechanism of interdisciplinarity arises very frequently indeed in relation to impacts of mathematics research: in interviews; in IETs showing 83% of units reporting it; and in surveys, with 82% of HoD respondents selecting it from a list. When asked specifically, most HoDs (86%) believed that ′active ‘collaboration with other disciplines has been a useful pathway in the generation of some of the impacts from my department′s research’. Almost two-thirds (64%) have seen results of ‘departmental research “picked up” later by other disciplines which then generate impacts’.

In contrast, only 35% of the ICSs explicitly referenced interdisciplinarity, perhaps the general tendency to omit mechanisms was amplified by a perceived risk in sharing credit with another discipline.

3.6.2 Relationships, relationship building

Interviews, surveys, and IETs gave a strong and consistent message about the importance of growing and sustaining long-term trusted relationships. Examples of remarks include these by two interviewees:

‘It [impact generation] usually comes down to having trusted someone in the past and asking their advice. In maths it is very much a personal thing. It is going to be a personal connection rather than someone reading a paper. People don’t think of mathematics as social but in this it very much is.’

‘It is critical but often best arrives as a result of a slow build-up of a relationship with industry colleagues who come to trust you and are prepared to share some data, if the CEO allows.’

Over two-thirds (68%) of HoD respondents selected ‘informal relationships, informal knowledge exchange’ as effective in helping to generate impacts, and when asked specifically about relationships, almost all (91%, with half of these strongly agreeing) believe that, in their departments, ‘those academics who have developed lasting relationships with individual stakeholders have generated the most impacts’.

Similarly, highlighting the role played by long-term relationships with particular stakeholders, half (50%) of the IETs mentioned long-term relationships with the unit; a third (35%) noted long-term relationships with an individual. For example:

‘Long-term relationships bring increased value to both sides: academics develop familiarity with the needs of collaborators, who, in turn, appreciate the value of the contribution that academics can make. The mechanisms used to sustain and develop a relationship are necessarily bespoke.’

Many IETs referred to the binding force of stakeholder-sponsored PhD studentships. Perhaps most intriguing are explicit references to one stakeholder relationship building upon another, seen in nearly two-thirds (63%) of the IETs. For example, one unit described steps including:

‘initial contact with new users’; ‘engaging with users to develop impact from current or pre-existing research’; and ‘developing and deepening relationships with key partners’.

Describing activities such as annual industry-outreach events and industry/academia workshops, yet another unit stated, with examples including a subsequent major consultancy contract and joint funding of a postdoctoral fellowship:

‘these activities are used as a platform to build longer term relationships with industry which are beneficial to both sides and generate impact’.

Again, in stark contrast to reflective discourses, and to 68% of HoD respondents, informal knowledge exchange was only described explicitly in 6% of the REF ICSs.

3.6.3 Knowledge intermediaries

While in both academic and practice-based discussions of knowledge exchange, ‘knowledge intermediaries’ are frequently seen as helpful, they have a low profile in mathematics. Just a 10th (9%) of HoD respondents selected the role from a list of mechanisms. In a later question about knowledge intermediaries specifically, only just over a quarter (27%) of respondents thought they played a useful role in helping to generate impacts and over a third (36%) thought they did not, with the rest neutral. It is possible that HoDs interpreted this question as referring in particular to staff in Central University knowledge transfer organizations, concerned with standard IP exploitation through patents, licences, and spin-outs, in which case their thinking correlates with the low profile of these impact mechanisms.

However, even extending the definition to include structural mechanisms such as industry consortia, we found the role mentioned in only a little more than a third (37%) of REF ICSs, and just over half (56%) of IETs. It is possible that some of these activities were present, while not explicitly mentioned, in other mechanisms such as ‘joint funding’ or ‘key responsibilities’ within departmental structures (often described in standardized ‘corporate’ language).

3.6.4 Culture

We suggest that the departmental (or university) culture within which academics work can play the role of a mechanism. Half (50%) of HoD respondents, split almost evenly between strongly agreeing and agreeing, believed that their ‘department/college/or university has provided a context or culture which deliberately facilitates the generation of impacts from mathematics research’, although nearly a quarter (22%) did not. There is clearly variation in the degree to which a unit makes use of multiple mechanisms (Table 6, including counts from Tables 1 and 2). Just four units (8%) utilized 20 or more mechanisms. Including these, over a third (37%) used 15 or more mechanisms.

Table 6.

Use of multiple mechanisms

Number of mechanisms used 20 or more 15–19 10–14 5–9 0–4 
Number of units of assessment 15 16 15 
Number of mechanisms used 20 or more 15–19 10–14 5–9 0–4 
Number of units of assessment 15 16 15 

While even in the IETs, explicit references to the intangible mechanism of deliberately fostering a ‘culture’ conducive to knowledge exchange were relatively infrequent (19%), it could be argued that units employing 15 or more mechanisms were in effect creating such a culture.

In addition to activity-based mechanisms such as workshops, IET analysis uncovered a number of organizational and administrative mechanisms which could contribute to a facilitative culture, such as inclusion in promotion considerations (Table 2).

4. Discussion

4.1 Bringing it together

Very often findings from different sources corresponded well with each other. When they did not, as was often the case with information ‘missing’ in the REF ICSs, the lack of alignment is itself of interest. In this discussion, we will focus on:

  • types of impact;

  • portfolios of mechanisms creating a facilitative culture for impact generation;

  • selected mechanisms—two prevalent mechanisms of relationship building and interdisciplinarity and two infrequently cited mechanisms of knowledge intermediaries and intellectual property protection; and

  • dynamics and non-linearity.

4.2 Utility of expanded definitions of impacts

The REF assessment places explicit value on contributions made by research beyond academia. Commendable in itself, this has created a large repository of impact narratives. The most common impact type by far was instrumental impacts. However, the full extent of impacts is even more impressive, as it is clear that many additional impacts arose from mathematics research but were not submitted to the REF.

Understandable as it may be for the first round of an exercise seeking consistency and transparency, the format of the ICSs and the nature of the impacts sought were relatively constrained. Subject-specific written guidance was provided on acceptable impacts which seems broader in some fields, e.g. Business and Management, than it is for natural sciences/mathematics, and was certainly interpreted more narrowly by those deciding on mathematics submissions.

This has curtailed expression of a full range of impacts; it would be unfortunate if this in turn curtailed recognition of the legitimacy of diverse impacts—with, potentially, correspondingly limited efforts taken to achieve the full range of impacts.

We saw this tension in the acknowledgement by HoD respondents that the number and diversity of impacts of mathematics research surpassed those instrumental impacts submitted to the REF. Many interviewees responded enthusiastically to our study’s ‘legitimisation’ of seemingly intangible impacts, often conveying a sense of regret and/or frustration at the perceived need to employ tactics which excluded types of impacts they felt to be important but ‘inapplicable’. Uncertainty existed as to acceptability for the REF of some sorts of conceptual impacts, seen as frequent and important for this underpinning field, but appearing less often in submissions. Thinking tactically, an HoD interviewee observed:

‘Conceptual impacts—new ways of thinking about a problem may have much greater impact down the line. The problem is the definition of impact we were given (for REF) was so narrow; we could never have put in a conceptual impact; it would never have passed muster. A lot of mathematicians were worried. The conceptual side of things is not nearly valued as much as it should be.’

Even sharper differentials for other types of impacts underscore the tactics inherent in drafting submissions. Over three quarters of HoD respondents saw capacity-building impacts but only just over a quarter submitted them. Offered the opportunity to consider the achievement of ‘enduring connectivity’, HoD respondents replied very positively in terms of having seen it, although very few had included anything of this sort in their submission. It has been argued elsewhere (Meagher and Lyall 2013) that this process-based impact has value in itself, while also potentially serving as a ‘proxy’ for enhanced likelihood of future more tangible impacts, given that researchers and stakeholders are involved in follow-on interactions and thus flow of knowledge or influence. The frequency with which HoD respondents see this process-based impact fits well with what emerged in this study as the central importance of relationship building as a mechanism. A non-academic stakeholder declared:

‘Long-term relationships are important. … just throwing research problems over the wall to people and having them throw the answer back is not effective.’

An overview interviewee captured both benefit and cost of connectivity:

‘There is an element of trust built up so both parties understand the interests of the other, where their strengths lie. … when an opportunity does arise, you already have a relationship and can exploit it quickly. My sense is that that is quite hard for universities to do and look after this, because it does take some resource for no immediate return at all.’

Despite attitude or culture changes seeming intangible, such impacts were seen by half of the HoD respondents, although under a quarter included these in their ICSs, and even some of these might have been changes in public attitudes, perhaps through outreach. Attitude or culture changes of the sort we had in mind can lead towards later impacts, described vividly by an interviewee recalling the long-term impact of an internship:

‘Where senior partners are aware that mathematics has helped their company in some way, then (they get) to the stage where when they have the next hard problem: they think ‘maybe we should talk to a Mathematician’. That is a culture change.’

We found public engagement or outreach lying somewhere between an impact (as often couched in REF ICSs) and an activity that might contribute towards an impact (as many would see it). Around a fifth of the ICSs included some form of public outreach (just seven were outreach only), while there were much higher percentages associated with public engagement as a mechanism in HoD responses and nearly three quarters of IETs, not constrained to link such activity to departmental research. The ambiguous and sometimes dual role of public outreach as impact or mechanism is a challenge for evaluation generally.

4.3 Mechanisms

4.3.1 Variable recognition of mechanisms

Recognition of roles played by various mechanisms in the generation of impacts is variable. Interviewees suspected that routes to impacts would be given short shrift in brief REF ICSs, constrained to telling a causal narrative linking academic articles with impacts, despite interviewees’ awareness that impacts tend to come about through activity such as interactions, collaborations, or contract research with stakeholders. One overview interviewee observed:

‘Case studies do not provide the scaffolding or the processes. … the formula the case studies had forced on them is not the way impact usually happens. … But the REF is only interested in the more linear process.’

Indeed, the recent independent review of the REF (Stern 2016: 17) notes as an issue that ‘the requirement to link ICS to key research outputs has meant that potentially very valuable channels whereby the UK's research base impacts on industry, public engagement and policy advice are not being captured’. The (near) absence of mechanisms in ICSs leaves by default a narrative in which one or two academic articles appear to lead in a straight line, as if inevitably, to impacts. There is perhaps a danger in this, if a complex dynamic appears falsely simple and units or indeed individual academics fail to perceive need for proactive efforts (mechanisms) to facilitate generation of future impacts.

4.3.2 Culture and portfolios of mechanisms

HoD respondents and IETs shed light upon the frequency with which different mechanisms were used. All but two UoAs noted five or more mechanisms in their IETs; over a third used 15 or more mechanisms. Whether the use of numerous mechanisms was a deliberate strategy towards robust impact generation, or an extensive list recognized retrospectively in writing the IET, it seems logical that a diverse portfolio of multiple mechanisms would make it more likely that multiple academics and multiple stakeholders would be influenced, along with the intangible ‘culture’ of a unit. Indeed, use of multiple mechanisms seemed to correlate broadly with the size and ambition of the unit.

Each UoA has its own ‘persona’; the portfolio of mechanisms it employs is distinctive in size and composition and the way in which any one mechanism is implemented. Yet, to gain a picture of what a proactive unit’s portfolio might look like, we gather here those 10 mechanisms employed by all four of the units citing the highest number of mechanisms (20 or more) in their IETs:

  • Long-term Relationship(s) of the department;

  • Building of relationships from other relationships;

  • Knowledge Intermediaries;

  • Other visits;

  • Events (more than one-offs);

  • Preparation of early career researchers;

  • Promotion/appraisal;

  • University structure/resources;

  • Research funding by stakeholders/joint funding;

  • Use of free software and other web-based materials to encourage uptake; and

  • Public engagement.

  • This core set reinforces our observations on the central importance of relationships, along with ways of building them.

Of course not all departments make such a concerted effort. Ambivalence can be seen in percentages of units that do or do not mention explicitly career incentives in IETs. Half cite consideration of knowledge exchange/impact generation in promotion/appraisal decisions, with two-fifths citing it for appointment/recruitment decisions; just over half allow for related activity in workload allocation and slightly over a third reward it. Whether these figures represent a cup half full or half empty may only be resolved by a later snapshot in time, exploring what might (or might not) become deeper embedding of impact-related activity in mathematics departments. Interestingly, much of the advice offered by interviewees regarding impact generation pertained to facilitative cultures. Still, cultural or attitude issues can arise regarding the generation of impacts, as an overview stakeholder interviewee describes:

‘When mathematics makes an impact, it can be in two ways: one, a new novel thing that is really useful mathematically. That is nice; a mathematician can publish and feel it is a huge advance in the field. Other times, the mathematics required is a bit more dirty, the black sheep in the family. … The clean beautiful maths or slightly dirty maths—it is very rare that they come together, although sometimes they do.’

4.3.3 A key mechanism: interdisciplinarity

Interdisciplinarity as a process emerged as a key mechanism for impact generation in mathematics. This is seen clearly in most survey responses and most IETs, and acknowledged very often by interviewees, perhaps especially in conceptual impacts on other fields.

This is consistent with a growing literature about an interwoven relationship between interdisciplinarity and impacts on complex problems—for example, natural and social scientists collaborating (Lowe, Phillipson Wilkinson 2013), ‘participatory interdisciplinarity’ towards knowledge production (O’Brien, Marzano and White 2013), and knowledge exchange involving multiple stakeholders and disciplines (Fazey et al. 2014). References to both the processes of interdisciplinarity and knowledge exchange in the context of impacts are included but not limited to what is frequently called ‘transdisciplinarity’ (Lawrence and Després 2004), Lawrence (2015); although UK research funders do not always use the term, they share this dual aim for tackling complex societal problems (Lyall, Meagher and Bruce 2015). A chapter devoted to knowledge exchange within a book on interdisciplinarity (Lyall et al. 2011) suggested that not only can interdisciplinarity contribute to knowledge exchange, but also that the processes share key features, including trust building.

A thought-provoking distinction among types of interdisciplinarity emerged from interviewees. Most academic literature focuses on ‘simultaneous’ interdisciplinarity, either within one multifaceted person’s approach or more often within a collaborative team. Yet, in addition, it seems that mathematics quite often gives rise to impacts via journeys through another discipline, what might be thought of as ‘sequential interdisciplinarity’, sometimes without direct interaction.

Interviewees expressed frustration that impacts via other disciplines were difficult to evidence, and not incentivized by the REF. If submission tactics (‘what counts’) were to be confounded with ‘what is important’, there could in the future be a danger of neglect for what has been very strongly articulated as a vital role played by interdisciplinarity in the generation of impact.

4.3.4 Key mechanism: relationship building

Relationship building is vital: informal relationships and informal knowledge exchange are cited by just over two-thirds of respondents, and building of relationships from other relationships is explicitly described in just under two-thirds of IETs, with interviewees often echoing this emphasis. In sharp contradistinction, such informal relationships appear in only 6% of ICS. This was not what the format required, as one HoD interviewee observed pragmatically:

‘You will find relationships airbrushed out of impact case studies, for sure. … For purposes of REF, there is not so much of the ‘how’.

Some IETs provide useful windows into the intangible but powerful role of relationships, and the use of a variety of mechanisms to develop and sustain them; for example, one institution cited: (1) a close collaboration of an academic and a PhD student with a company, follow-up, industry workshops, and on-site presentations at various companies; (2) collaborative industry research helped by an industry-funded lectureship, specific industry workshops, sabbatical leave, and consultancy; (3) a long-lived interdisciplinary and industry collaboration aided by particular hires, incoming secondments from industry, and outward sabbatical leaves; and (4) triggering a long-term collaboration with teaching and regular research visits.

Informal and intangible as human interactions are, they are seen to make a difference. Sustained interactions with nodes of expertise lead to long-term relationships with external research users, creating trust, and understanding which can then be brought to bear on problems as they arise. This is consistent with the 2015 survey (Hughes et al. 2016), which found the most common forms of external knowledge exchange to be non-commercial people-based, problem-solving and community-based interactions, which had been sustained since the previous 2008 survey, whereas commercial activity (licencing, etc.) had declined.

4.3.5 Infrequently cited mechanisms: formal KE activity and knowledge intermediaries

Two mechanisms generally regarded as key to impact generation were cited surprisingly infrequently in mathematics ICS: the role of knowledge intermediary, and classic ‘technology transfer’ through IP exploitation, patents, spin-outs, and licences.

The latter is consistent with assertions made by learned societies to government (CMS 2015), as well as other analyses of ICSs. EPSRC’s analysis of spin-out data corroborated our analysis, to show a modest number of seven spin-outs reported for mathematics, by far the smallest number among EPSRC disciplines (EPSRC 2015).

Interviewees suggested that the generic, open, and interdisciplinary nature of mathematical research means that it often underpinned work in other disciplines that then led to exploitable IP, with the link to the underlying mathematics being lost, as this overview interviewee captures:

‘Once you’ve seen an algorithm work in a particular situation, you just apply it somewhere else and don’t even reference it. It is not generally the culture of mathematicians to establish IP rights. So there are many situations where you do not have a formal IP connection.’

Interviewees described how mathematicians often lay the groundwork for others such as engineers to later create inventions; one overview interviewee saw this as ‘sequential interdisciplinary’ work taking place at the ‘boundary between conceptual and instrumental impacts’.

This helped us clarify a puzzling dichotomy. On the one hand, our surveys and interviewees rated knowledge intermediaries of little relevance; on the other hand, deeper digging uncovered a variety of examples of individuals and structures fulfilling the role, through references to the importance of building long-term relationships, industry consortia (seen in around a third of ICSs) and, with a broad working definition of knowledge intermediary, evidence seen in slightly over half of IETs.

The dichotomy is resolved on realizing that many of our informants were probably interpreting the term narrowly, as technical specialists employed by universities (or related bodies) to support IP generation, which is indeed of lower relevance to mathematics. Caveats run through many interviews, for example this overview interviewee comment:

‘There are people paid by the research office but we tend to see them as administrators to get off our backs—“what do they know about this stuff?”’

On further probing, interviewees recognized the importance of the broader role of a knowledge intermediary, drawing on bridge-building metaphors, stressing commitment, and enthusiasm.

Widening the definition of Knowledge Intermediary to structural entities included a number of valued organizations, for example the Industrial Mathematics KTN (formerly the Smith Institute), referred to in the IETs of essentially the nine largest submissions. This currently runs the highly regarded ‘Study Groups with Industry’ (ESGI 2016), started in 1968, that brings together academics and industry in a carefully structured 5-day format where representatives from industry present problems and work with mathematicians to brainstorm ideas and work towards practical solutions, subsequently developed over following weeks. Such ‘structures’, as well as sustained efforts by learned societies and other organizations, bring academic and non-academic researchers together often over a sustained period of time, allowing them to build relationships and trust.

A newly emerging knowledge intermediary is research software, identified in one in six case studies. UK universities have been responsible for a number of influential and widely used software packages. In addition, making research techniques available as free software (often open source, so it can also be freely modified), which can be downloaded and tried out, while shielding users from the technicalities of the underlying mathematics, is becoming increasingly important for opening up mathematical ideas to new domains.

4.3.6 Dynamics and non-linearity of impact generation

Nearly all HoD respondents are firmly of the belief that ‘in general, a long timeframe elapses between mathematics research and the development of impacts’. This sense of time is consistent not only with examples provided in our interviews and focus group, but also with research into conventions of mathematical researchers, showing that patterns of citation in mathematics are longer than in other science disciplines (Adler et al. 2009), with mathematicians appearing to prefer original sources to more recent alternatives, in an approach closer to the humanities (Ferrer-Sapena et al. 2016). Furthermore, two time-consuming dynamics of mathematics research can prolong the timeline until impact: relationship building and interdisciplinarity.

Impact generation is rather more multidimensional than linear. Close reading of groups of case studies allows one to build an overall picture, seen through multiple small uniform windows, of rich, long-lived, complex, and highly interlinked ecosystems, where particular individuals, techniques, software tools, research consortia, and partners in the public and private sectors reappear in different case studies. So, for example, we examined the 43 ICSs which referred to Bayesian statistical methods. These narrated, as required by the format, 43 chains of impact from research paper to research user, but, when considered as a whole, depict something much richer—a teeming ecosystem of activities from which the interrelated chains have been extracted. This rich picture meshes with the reflections of HoD interviewees, referring to a current ‘hot topic’ in the era of big data:

‘In mathematics it is a much longer chain (to impact) in many ways. … The chain may not be seen ahead of time, maybe with hindsight you see there was a chain there. But maybe ‘chain’ is the wrong word. … For instance, one type of topology is quoted quite a lot as being relevant, persistent homology. … Persistent homology was developed by many people, not one. … Quite often that is the way it works in mathematics, with lots of people working on it. If you try in fifty years to look back and see a chain, you would be inventing a story after the fact rather than it being the way it really happened.’

4.3.7 Looking to the future

Mathematics research has indeed generated a diverse range of impacts, through richly textured and complex mechanisms not captured readily in a single assessment framework. Interviewees, who play leadership and/or overview roles, engaged with the aim of this investigation, placing value on understanding of how mathematics generates impacts and on validation of conceptual or other ‘indirect’ impacts, particularly as mathematicians and departments look ahead to the next REF and the ongoing impact agenda. An HoD interviewee captured this:

‘I believe that if people see a mechanism or framework for getting to impact, then it becomes a task; you have a compass and you can get there. This time, we did not know. … So, if this study provided a clear statement about what impact is for mathematics, how one achieves it and how that could be supported by our universities, the mathematics community would feel a lot more engaged with the process.’

5. Conclusion

5.1 Multiple methods and multidimensional understanding

The triangulated approach of this empirical study made it possible to illuminate ‘how’ mathematics research has led to impacts. This has shed light upon the ‘triple conundrum’ posed earlier, as external assessment restrictions, along with cultural traditions reacting against a limited set of traditional metrics, have obscured the full range and dynamics of influences permeating society. Close analysis of the REF ICSs provided a snapshot of impacts (primarily instrumental). This picture was fleshed out considerably by qualitative methods (surveys, interviews, and focus group), which uncovered far more impacts. Opening up the dialogue to include less tangible impacts coincided with consideration of a number of subtle mechanisms (such as relationship building or interdisciplinarity) involved in flows of knowledge. Content analysis of IETs benefited from units expounding upon mechanisms. We found that this multi-method approach gave us both balance and the opportunity to unveil key aspects of underlying processes.

Surveys, interviews, and IET analysis all underscore a range of subtleties. Changes in conceptual understanding; human interactions; building relationships upon relationships; being flexible, opportunistic, and responsive as new problems arise—all this is a much more complex, diffuse, and intangible picture than that which is asked for by or reported to the REF. Whatever the cause—a laudable drive for comparability by the REF or adherence to space constraints and presumed winning tactics by units submitting case studies—an artificially simple vision for the generation of impact is in danger of being reified. While the case studies selected, crafted, and submitted to the REF are undeniably compelling, it would be unfortunate if these tactics so narrowed the view of impacts, and so definitively assumed linear causality between research papers and impacts, that diversity of impacts and subtleties of important processes were ignored going forward. Ironically much will have been lost rather than gained by the recent emphasis on impact assessment of a particular kind.

Instead, this study’s close examination of processes connecting research and impacts in one academic field, mathematics, provides a more multidimensional view of how knowledge flows. We hope it will contribute to an evidence base that can help researchers and research leaders in various fields appreciate more fully diversity among types of impacts and among mechanisms with potential for impact generation. Similarly, provided the long-term and contingent nature of impact generation is recognized, a heightened awareness of a dynamic system of processes and impacts can contribute towards capture of progress indicators, in turn enhancing prospects for later impact.

5.2 Concluding suggestions

We have taken a ‘zoom lens’ into the relationship between research and impacts in mathematics, but we now pull back our focus and make suggestions to stimulate thinking about that relationship more broadly.

Recognition of breadth of impacts Mathematics research has given rise to numerous impacts, demonstrating an array far wider than that captured in REF ICS and highlighting in particular the crucial role of conceptual impacts. For all fields, there is a danger that, if only ‘what counts’ is recognized, important impacts and indeed process-based impacts that may represent steps towards desired tangible impacts may not be pursued.

Recognition of breadth of mechanisms Diverse mechanisms have contributed to the generation of impacts from mathematics, commending to any field a robust ‘portfolio’ of mechanisms. Individuals, centres, and departments/units of assessment can all take proactive steps to enhance likelihood of future impacts, including but not limited to creation of facilitative cultures.

Complexity and dynamics Recognition that impacts develop within dynamic ‘ecosystems’ over time can sharpen awareness of both stage-appropriate steps and ways to identify and build upon key points along the journey.

Relationship building A central mechanism (incorporating various efforts and activities) is relationship building. Strong long-term relationships between researchers and ‘stakeholders’ frequently appear to be critical to the generation of impacts. Recognition of this requires a narrative more subtle than a ‘one research paper gives rise to one impact’ story line. Implementation requires awareness of ways in which informal relationships can be created and maintained.

Interdisciplinarity Interdisciplinarity is a mechanism or process leading towards impacts that may be particularly vital to some areas of research, such as mathematics, but also, more generally, to increasingly complex societal problems. Recognition of the role that interdisciplinarity can play in generating impacts may inform future behaviours (and assessments).

Knowledge intermediaries A conventional institutional knowledge intermediary will not always be the most effective. The valuable role of knowledge intermediary can take on diverse forms, filled by variously placed individuals or structures (e.g. industry consortia, centres or even free software). A committed and trusted academic may often directly perform this function.

Facilitating multidimensional impacts Overall, there is a dynamic richness to the multidimensional picture that emerges, but it is neither impenetrable nor a barrier to action. Indeed, we suggest that opening eyes to diverse impacts, and to the many mechanisms which can lead to their development over time, will enhance the likelihood of meaningful impacts being generated from research in the future.

Acknowledgements

We acknowledge support from EPSRC under grant EP/K040251, an EPSRC Fellowship awarded to the second author to study the mathematical research process. We thank our informants for their thoughtful responses in interviews, questionnaires and a focus group; EPSRC for early access to data; the Heads of Departments of Mathematics for allowing us to present at their annual meeting in 2015; Catherine Lyall, Peter Mcowan, Tom Meagher and Sandra Nutley for insightful comments; and Tom Meagher and Jessica Meagher for assistance with data analysis.

Funding

This work was supported by the Engineering and Physical Sciences Research Council under grant EP/K040251/2.

References

Adler
R.
Ewing
J.
Taylor
P.
(
2009
) ‘
Citation Statistics: A Report from the International Mathematical Union (IMU) in Cooperation with the International Council of Industrial and Applied Mathematics (ICIAM) and the Institute of Mathematical Statistics (IMS)
’,
Statistical Science
 ,
24
(
1
):
1
14
.
Armstrong
F.
Alsop
A.
(
2010
) ‘
Debate: Co-production can Contribute to Research Impact in the Social Sciences
’,
Public Money & Management
 ,
30
:
208
10
.
Bannister
J.
Hardill
I.
(
2013
) ‘
Knowledge Mobilisation and the Social Sciences: Dancing with New Partners in an Age of Austerity
’,
Contemporary Social Science
 ,
8
:
167
75
.
Boaz
A.
Locock
L.
Ward
V.
(
2015
) ‘
Whose Evidence is it Anyway?
’,
Evidence and Policy: A Journal of Research, Debate and Practice
 ,
11
:
145
8
.
Bryman
A.
(
2012
)
Social Research Methods
 ,
4th edn,
Oxford
:
Oxford University Press
.
Bush
V.
(
1945
)
Science: The Endless Frontier: A Report to the President on Post-war Scientific Research
 ,
Washington, DC
:
US Government Printing Office
.
Cameron
P.
(
2011
), ‘
LMS Impact Consultation
’,
Peter Cameron's Blog
 , <https://cameroncounts.wordpress.com/2011/05/20/lms-impact-consultation/>.
CMS
(
2015
), ‘CMS Response to the Dowling Review of Business-University Research Collaborations’, <http://www.cms.ac.uk/files/Submissions/article_553915d5703789.23364772.pdf>.
Cooper
A.
(
2014
) ‘
Knowledge Mobilisation in Education Across Canada: A Cross-case Analysis of 44 Research Brokering Organisations
’,
Evidence & Policy: A Journal of Research, Debate and Practice
 ,
10
:
29
59
.
Davies
H.
Nutley
S.
Walter
I.
(
2005
), ‘Approaches to Assessing the Non Academic Impact of Social Science Research’, in
ESRC Symposium on Assessing the Non Academic Impact of Research 12/13 May 2005
 ,
St Andrews
:
School of Management, University of St Andrews
.
Davies
H.
Powell
A. J.
Nutley
S.
(
2016
), ‘Mobilizing Knowledge in Health Care’, in
Ferlie
E.
Montgomery
K.
Pederson
A. R.
(eds)
Oxford Handbook of Health Care Management
 ,
Oxford
:
Oxford University Press
.
Deloitte
(
2013
), ‘Measuring the Economic Benefits of Mathematical Science Research in the UK’, <http://www.ima.org.uk/_db/_documents/Deloitte_report.pdf>, p. 37.
EPSRC
(
2015
), ‘Investing in Excellence, Delivering Impact for the UK, Insights from the Research Excellence Framework 2014’, <https://www.epsrc.ac.uk/newsevents/pubs/refreport2015/>.
ESGI
(
2016
), ‘ESGI116, University of Durham’, <http://www.esgi.org.uk/>.
Fazey
I.
et al.  . (
2014
) ‘
Evaluating Knowledge Exchange in Interdisciplinary and Multi-stakeholder Research
’,
Global Environmental Change
 ,
25
:
204
20
.
Ferrer-Sapena
A.
et al.  . (
2016
) ‘
The Impact Factor as a measuring tool of the prestige of the journals in research assessment in mathematics
’,
Research Evaluation
 ,
25
(
3
):
306
13
.
Grant
J.
et al.  . (
2010
), ‘Capturing Research Impacts. A Review of International Practice’, Prepared for HEFCE by RAND Europe, <http://www.rand.org/content/dam/rand/pubs/documented_briefings/2010/RAND_DB578.pdf>.
Greenhalgh
T.
Fahy
N.
(
2015
) ‘
Research Impact in the Community-based Health Sciences: An Analysis of 162 Case Studies from the 2014 UK Research Excellence Framework
’,
BMC Medicine
 ,
13
/
1
:
232
.
Hardy
G. H.
(
1940
),
A Mathematician's Apology
 ,
Cambridge, England
:
The University Press
.
Hughes
A.
et al.  . (
2016
),
The Changing State of Knowledge Exchange: UK Academic Interactions with External Organisations 2005 -2015
 ,
(London: NCUB)
.
Jagosh
J.
et al.  . (
2015
) ‘
A Realist Evaluation of Community-based Participatory Research: Partnership Synergy, Trust Building and Related Ripple Effects
’,
BMC Public Health
 ,
15
/
1
:
725
.
King's College London and Digital Science
(
2015
),
The Nature, Scale and Beneficiaries of Research Impact: An Initial Analysis of Research Excellence Framework (REF) 2014 Impact Case Studies
 ,
Bristol, UK
:
HEFCE
.
Kothari
A.
et al.  . (
2011
) ‘
Indicators at the Interface: Managing Policymaker-Researcher Collaboration
’,
Knowledge Management Research and Practice
 ,
9
/
3
:
203
14
.
Lawrence
R. J.
(
2015
) ‘
Advances in Transdisciplinarity: Epistemologies, Methodologies and Processes
’,
Futures
 ,
65
:
1
9
.
Lawrence
R. J.
Després
C.
(
2004
) ‘
Futures of Transdisciplinarity
’,
Futures
 ,
36
/
4
:
397
405
.
Lightowler
C.
Knight
C.
(
2013
) ‘
Sustaining Knowledge Exchange and Research Impact in the Social Sciences and Humanities: Investing in Knowledge Broker Roles in UK Universities
’,
Evidence & Policy: A Journal of Research, Debate and Practice
 ,
9
/
3
:
317
34
.
LMS
(
2011
), ‘Impact in the mathematical sciences in REF2014’, www.lms.ac.uk/sites/lms.ac.uk/files/Mathematics/policy_responses/RPCimpact8.pdf
Lomas
J.
(
2000
) ‘
Using ‘Linkage and Exchange’ to Move Research into Policy at a Canadian Foundation
’,
Health Affairs
 ,
19
/
3
:
236
40
.
Lowe
P.
Phillipson
J.
Wilkinson
K.
(
2013
) ‘
Why Social Scientists should Engage with Natural Scientists
’,
Contemporary Social Science
 ,
8
/
3
:
207
22
.
Lyall
C.
Meagher
L. R.
Bruce
A.
(
2015
) ‘
A Rose by any Other Name? Transdisciplinarity in the Context of UK Research Policy
’,
Futures
 ,
65
:
150
62
.
Lyall
C.
et al.  . (
2011
),
Interdisciplinary Research Journeys: Practical Strategies for Capturing Creativity
 ,
1st edn
, p.
240
.
London
:
Bloomsbury Academic
.
Meagher
L. R.
Lyall
C.
(
2013
) ‘
The Invisible Made Visible: Using Impact Evaluations to Illuminate and Inform the Role of Knowledge Intermediaries
’,
Evidence and Policy
 ,
9
/
3
:
409
18
.
Meagher
L. R.
Lyall
C.
Nutley
S.
(
2008
) ‘
Flows of Knowledge, Expertise and Influence: A Method for Assessing Policy and Practice Impacts from Social Science Research
’,
Research Evaluation
 ,
17
/
3
:
163
73
.
Molas-Gallart
J.
Tang
P.
(
2011
) ‘
Tracing ‘Productive Interactions’ to Identify Social Impacts: An Example from the Social Sciences
’,
Research Evaluation
 ,
20
/
3
:
219
26
.
Molas-Gallart
J.
Tang
P.
Morrow
S.
(
2000
) ‘
Assessing the Non-academic Impact of Grant-funded Socio-economic Research: Results from a Pilot Study
’,
Research Evaluation
 ,
9
/
3
:
171
82
.
Nature Outlook
(
2014
), ‘
Assessing Science: Lessons from Australia and New Zealand
’,
Nature
 ,
511
/
7510
:
S49
83
.
NRC
(
2014
),
RF Celeste, A Griswold, and ML Straf, Furthering America's Research Enterprise
 ,
Washington, DC
:
The National Academies Press
.
Nutley
S.
Walter
I.
Davies
H. T. O.
(
2007
),
Using Evidence: How research can inform public services
 ,
Bristol
:
The Policy Press
.
O’Brien
L.
Marzano
M.
White
R. M.
(
2013
) ‘
Participatory Interdisciplinarity’: Towards the Integration of Disciplinary Diversity with Stakeholder Engagement for new Models of Knowledge Production
’,
Science and Public Policy
 ,
40
/
1
:
51
61
.
Phillipson
J.
et al.  . (
2012
) ‘
Stakeholder Engagement and Knowledge Exchange in Environmental Research
’,
Journal of Environmental Management
 ,
95
/
1
:
56
65
.
Powell
A. J.
Davies
H.
Nutley
S.
(
2016
) ‘
Missing in Action? The Role of the Knowledge Mobilisation Literature in Developing Knowledge Mobilisation Practices
’,
Evidence and Policy, Fast track articles
 , 22.01.2016.
RCUK
(
2007
), ‘Excellence with Impact’, <http://www.rcuk.ac.uk/documents/economicimpact/excellenceimpact-pdf/>.
Reed
M. S.
et al.  . (
2014
) ‘
Five Principles for the Practice of Knowledge Exchange in Environmental Management
’,
Journal of Environmental Management
 ,
146
:
337
45
.
REF UK
(
2014
), ‘REF2014’, <http://www.ref.ac.uk>.
Roux
D. J.
et al.  . (
2010
) ‘
Framework for Participative Reflection on the Accomplishment of Transdisciplinary Research Programs
’,
Environmental Science & Policy
 ,
13
/
8
:
733
41
.
Samuel
G. N.
Derrick
G. E.
(
2015
) ‘
Societal Impact Evaluation: Exploring Evaluator Perceptions of the Characterization of Impact Under the REF2014
’,
Research Evaluation
 ,
24
/
3
:
229
41
.
Spaapen
J.
van Drooge
L.
(
2011
) ‘
Introducing ‘Productive Interactions’ in Social Impact Assessment
’,
Research Evaluation
 ,
20
/
3
:
211
8
.
Steingart
A.
(
2013
), ‘
Conditional Inequalities: American Pure and Applied Mathematics, 1940-1975
 ’,
(Cambridge, MA: MIT)
.
Stern
N.
(
2016
), ‘Building on Success and Learning from Experience: An Independent Review of the Research Excellence Framework’, <https://www.gov.uk/government/uploads/system/uploads/attachment_data/file/541338/ind-16-9-ref-stern-review.pdf>.
Vogan
D.
(
2013
), ‘Written Testimony to US House of Representatives Committee Regarding FY2014’, <www.ams.org/policy/government/advocacy/FY2014HouseCJStestimony.Vogan.pdf>.
Wilsdon
J.
et al.  . (
2015
). ‘The Metric Tide: Report of the Independent Review of the Role of Metrics in Research Assessment and Management’, <http://www.hefce.ac.uk/pubs/rereports/Year/2015/metrictide>.
This is an Open Access article distributed under the terms of the Creative Commons Attribution License (http://creativecommons.org/licenses/by/4.0/), which permits unrestricted reuse, distribution, and reproduction in any medium, provided the original work is properly cited.