Affective auditing: The emotional weight of the research excellence framework

Abstract

In aggressively neo-liberalized higher education systems and in ‘high-performing’ research units—typically academic schools in high-ranking research universities—research assessment has come to dominate the daily organization and enactment of research and research culture. So much so in fact that academics’ research praxis, their employability, career trajectories and very lexicon are in synthesis with the manufacture and mediation of performance values, often to the detriment of collegiality, critical citizenship, and self-efficacy. Research assessment as a technology of governance is thus also a ‘disruptive technology’ epidemic to the (re)making of academic lives. Notwithstanding, studies of the affective aspects of research assessment and its emotional manipulation of academic lives are at best thin. Further, less is known of what we call ‘affective auditing’ from the perspective of academic middle-managers with institutional responsibility for implementing assessment procedures and with direct experience of the disruptiveness of research assessment at meso and micro levels. By way of response, this article reports on findings from interviews with academic middle or quasi-managers responsible for overseeing research assessment in research elite universities in the high-performance and highly pressurized research context of the UK. These accounts elucidate the weight of ‘affective auditing’ on academic researchers and academic quasi-managers and the extent to which research assessment shapes the emotional contours of research lives.

1. Introduction

The present ubiquity of an ‘audit culture’—and audit as a technology of governance (cf. Oancea 2019)—within universities may be explained as an effect of knowledge capitalism and a prerequisite of universities’ participation within a globally competitive prestige economy (Blackmore and Kandiko 2011). The proliferation of audit culture is also theorized as an antidote to concerns of societal ‘risk’ (Beck 1992) and corrective to a crisis of confidence in the self-regulation of public organizations, despite concerns that auditing may no more militate than escalate the ill-faith of external stakeholders, where appropriated, disingenuously, for purposes of positional gain (Watermeyer and Tomlinson 2022).

Auditing of academic research as a public investment, justifiably requiring in corollary, evidence of a public return, is seen to incentivize systemic game-playing and universities’ guile in the choreography and creative enumerations of their productive output; particularly as translates to quantitative declaration of excellence as the de facto barometer of public value. The distillation of research excellence into numerical indicators and performance indices—orchestrated through performance-based research funding systems, or more simply put, ‘audit technologies’, like the UK’s Research Excellence Framework (REF)—provides a vastly reductionist yet equally efficient means for universities to attest superiority in a field of equally superior competitors (cf. Olssen 2016; Collini 2017). Universities’ attempts for quantitative mesmerization have become integral to their efforts in securing public funds for research; in weaponizing against the anti-intellectual hostilities of populist politics and New-Right ideologists; in out-claiming their institutional competitors in the chase for accumulated capital, and as an endgame—or rather, zero-sum game—in which the working lives of academics are implicated. And despite the universalism of such a strategy across neoliberalized higher education domains, quantitative signification of excellence—even if underpinned by narrative depositions—remains route-one to preserving the munificence of universities’ public backers. It is also, despite the lamentations of academics apropos the surrender of self-governance to a ‘tyranny of metrics’ (Muller 2019), a ubiquitous and prominent feature of their self-presentation.

Academics may be observed as ‘auditees’ (Power 1994) imposed upon by, yet complicit within a continuous, complex and costly process of performance evaluation that is intended to both monitor and stimulate productivity, yet which also disrupts the rhythm of intellectual labour with managerialist incursions that systematically weaken professional autonomy and undermine collegiality and trust (cf. Watermeyer and Olssen 2016). It is in such terms that an obligation of public accountability provides a foil for the centralization of power within universities and the erosion of academics’ work-based freedoms and powers of dissent. Concurrently, where accountability has been confused with accountancy, ‘being answerable to the public’ is recast in terms of measures of productivity, ‘economic efficiency’, and delivering ‘value for money’ (Shore 2008: 281). Watermeyer (2019) theorizes that universities’ commitment to being publicly accountable has mutated into a self-serving exercise of ‘competitive accountability’, which sees academics’ expressions of public contribution, qualified through auditing processes, being exploited for their positional value and transformed into capital assets. It is the relative accumulation of these that we claim demarcates academics’ positionality within a performance hierarchy and thus relative share of academic capital, which by extension, determines the extent to which they may court institutional favour or censure, reward, or rebuke. A willingness and/or aptitude to be favourably measured is thus preconditional to academics’ navigation and survival of the precarity and phantasmagoria endemic to an ethos of competitive accountability. Yet, it is also emblematic of what Butler (2020: 21) describes as the ‘transformative effects of loss’—in this case as relates to the mourning of idealizations of academic identity—through which is revealed forms of attachment, relational ties for instance to the REF, that may be alien and antagonistic to self-concept, which may even be dehumanizing, yet which undergird the manifestation of organizational identity in the ‘audit society’ (Power 1997).

The impact of academics’ experience of auditing on their identity formation and public presentation is profound yet is ostensibly less well understood from the perspective of the emotional infrastructure in which it sits, and which the banality of numbering obfuscates. In this article, we aim to become better acquainted with the affective dimensions of auditing on academic lives by considering internally administered REF evaluation processes in UK universities as a series of emotional encounters, and potentially highly emotive events, involving academic evaluators and those whose work they evaluate. We hope to expose the emotional albatross endured, yet to date, undisclosed by academic quasi-managers leading departmental/(sub)disciplinary REF submissions through a series of n = 21 interviews, so as to further complicate a narrative of burden associated with the machinery of research evaluation. We introduce a concept of ‘affective auditing’ to describe how managerialist and surveillance cultures in universities and the practice of competitive accountability endemic to the application of technologies of research governance, influence and affect the emotional and ethical contours of academic work, habitually in negative ways. We suggest, correspondingly, that a human-centred conceptualization of research evaluation is especially urgent given concerns of labour precarization, intensification, exploitation, and attrition in universities escalated by the COVID-19 crisis and lamented upon as ‘pandemia’ (Watermeyer et al. 2021), in addition to calls for the fuller prioritization of health and wellbeing within organizational management and more compassionate and kinder research culture in universities (Derrick 2020). Our study of REF-Leads is also important in terms of further examining hybrid (Sirris 2019), insecure and/or contested identities precipitated by and ubiquitous within research audit regimes, and the challenge of leadership for those sceptical of both the utility value and social efficacy of evaluation technologies.

We begin by considering the role of ‘affect’ in making sense of academic lives, recognizing with Clegg (2013: 71) that ‘understanding how affect operates is central to understanding higher education’. Thereafter, we provide a full account of REF evaluation as an internal process within UK universities, for the benefit especially of an international readership less familiar with the meso-level institutional realities of life under REF as affective auditing.

1.1 Affect

We borrow from critical theorists like Sara Ahmed and Lauren Berlant in applying ‘affect’ to make sense of the emotional economy of academic life and its influence over the formation—and potential devastation—of research identities. We use emotions as an explanatory device that elucidate, ‘not only how we are affected in this way or that, but also how those judgements hold or become agreed as shared perceptions’ (Ahmed 2014: 208). Emotions both produce and are produced by the ‘affective atmosphere’ (Brennan 2004) of social worlds. In other words, we view emotions as simultaneously a form and consequence of affect.

We consider internal REF evaluation processes as producing a vortex of emotions that ‘create the very effect of the surfaces or boundaries of bodies and worlds’ (Ahmed 2004: 117). We view emotions, intrinsic to and produced through involvement in internal REF processes, as socially constructive and bringing academic bodies into being (cf. Gregg and Seigworth 2010). Emotions thus offer a window onto the academic self. We also understand emotions to be an iteration of capital, variations of which contribute to the unequal distribution of power within universities.

Our study of ‘affective auditing’ is also intended as further provocation of how the REF, as a case of ‘winners and losers in a game of academic prestige’ (Adler and Harzing 2009: 74) serves to both assemble and disassemble research identities. As Harley (2002) has noted, research selectivity exercises have high identity value, which may be attributed to the affective influence of peer judgements of disciplinary contributions to academic identity and self-esteem (Henkel 2000).

An idea of ‘cruel optimism’ coined by Lauren Berlant is also useful to our consideration of the role of affect and the REF as one particular affective economy. Berlant (2010: 94) describes cruel optimism as ‘attachment to compromised conditions of possibility whose realization is discovered either to be impossible, sheer fantasy, or too possible, and toxic’. In the REF, we visualize cruel optimism as the habituation of immiseration and academics’ (forced) compliance, despite a political economy of disavowal, to ‘ride the wave of the system of attachment they are used to’ (Berlant, 2010: 97). Cruel optimism thus provides a framing through which is made manifest the emotional injuries of affective auditing, and the REF as an object of desire (fear and hatred) that will be argued is detrimental to the flourishing of research identities and research communities in universities.

1.2 REF at the meso-level

Responsibility for the REF is in most UK universities overseen by someone at executive level, typically a Pro-Vice Chancellor (or equivalent) holding the research portfolio and supported by a professional services leader. Responsibility for the REF is further devolved and distributed at faculty and departmental levels. In the latter case, those who we designate in this article, REF-Leads—typically mid-to-late career academics with significant research experience—play a pivotal role in the collection of documentation, specific to their disciplinary unit of assessment that forms a part of a larger institutional submission or REF-return. The role of REF-Leads has been scarcely if at all studied and knowledge pertaining to their experience as stewards of the REF’s auditing process is predominantly anecdotal. Their study, however, is pertinent as significant actors within UK higher education’s prestige economy and individuals of complex, indeed, contested identity. Much as other academic managers, their identity is found to oscillate between an idealized version of their academic selfhood that objects to the intrusion of the REF as an instrument of neoliberal rationality, and a pragmatist version of their academic selfhood that accepts the REF as an inescapable feature or a necessary evil dominating all academic lives. It is as a necessary evil, that REF-Leads may attempt to justify their identity conflict and self-rationalize the compromise demanded in their leadership as audit managers, and the betrayal of ideals others may accuse them of.

While the REF is a process of external auditing, with submissions judged by expert panels of academics and user-assessors, it is for the most part an inherently institutional affair, and one rarely without intermission. Though institutional submissions occur periodically and in five- to seven-year intervals, internal evaluation of all REF-related activity, specifically the generation of research outputs and impact generative activities is continuous. Given the significance of the REF to universities in both monetary and prestige capture, a permanent REF infrastructure that corresponds to the need of universities to be on continuous REF-watch or as we prefer, REF-alert, is exigent. While the intensity of REF-alert fluctuates, research performance evaluation is ever-present and contributes to the pervasiveness of an audit and surveillance culture within UK universities. The intensity of the REF-Lead role—and also, identity conflict—is most acute, the closer to an institutional REF submission, and where internal auditing processes intended to curate the most competitive return of research outputs, impacts and environment statements, increase in regularity. At such points, the REF-Lead role may also change. In times of lesser intensity, further away from REF submissions, REF-Leads may enjoy a more explicitly collegial interface with colleagues, and a more facilitative and supportive role. Where universities, creep closer to institutional submissions, REF-Leads are forced to adopt a more authoritarian policing role, in leading their departmental colleagues towards a competitive return via strict auditing of their research outputs and impacts. Yet such leadership is a point of contention, where the authority of REF-Leads is largely symbolic and superficial, and while making powerful decisions—with significant consequences affecting research identities and general morale—they may be largely powerless to change the course of events. We comment here on how the vast majority of REF-Leads will be without line-management or budgetary powers, and therefore without capacity to either incentivize or facilitate strong research performance or else censure those academics who fail to fulfil research expectations (delineated by the REF). Self-evidently, they are not directly responsible for the failure of an academic colleague to deliver research outputs and impacts of REF standard,¹ however, they are indirectly, in so much as they assume overall ownership of a disciplinary submission. They are, therefore, despite claims to the contrary, heavily accountable for the ‘competitiveness’ of a unit of assessment, that might be marred by the inability of an academic colleague to fulfil on the REF’s requirement of research excellence.

While REF-Leads lack the kinds of structural power exercised by many academic managers, their affective influence over their immediate colleagues within the REF auditing process is substantial. They hold powerful knowledge related to the REF-ability of their colleagues, in terms of individual (and confidential) evaluation scores pertaining to research outputs and impact which are taken as a proxy for their departmental research contribution, and therefore status. Their impact on research identities, the self-efficacy and self-esteem of researchers, where imparting REF value-judgements, may be profound; especially when recognizing the frailty of self-concept common to many researchers. Interactions involving REF-Leads and academic colleagues may become especially challenging, where performance demands closer to REF submission points escalate and internal institutional scrutiny of Unit of Assessment (UoA) submissions intensifies. Audit focused interactions may vary not only according to relative levels of performance/productive output, but capacity for such influenced by a variety of professional and social determinants such as research experience and/or job role and contractual demands and gender. Interactions may thus differ wildly for instance, between a young female lecturer, with limited research experience and output track-record, and a senior male professor with a long track record of research publications. Of course, either could go either way, with an early career academic able to produce 4* REF outputs, while a senior academic having a yield close to none, resulting in very different kinds of conversations. Of greater symmetry and commonality across these interactions, is the emotional labour involved in what can be open and non-anonymous value-judgements of a colleague’s work.

While academic peer-review processes are normally anonymous, with the identities of authors and reviewers kept apart from either party, in the context of the REF, identities are revealed, at least those of REF-Lead and academic colleagues whose research is internally evaluated. Anonymity in peer-review is undertaken explicitly on the basis of a safe-check against (unconscious) bias, impartiality and/or prejudicial favours by reviewers when making value-judgements of scientific work. It is also implicitly undertaken as a matter of ethics and non-disclosure as a form of protection in the context of judgements that might cause disagreement and potential conflict. Of course, in the context of universities internal review at UoA levels, such anonymity is largely impossible, even where wider teams of peer-reviewers other than the REF-Lead are recruited. Authors will tend to be known to reviewers, and reviewers’ identities may be easily deduced based on matched expertise. Internal REF evaluation processes may, therefore, be susceptible to partisanship and decisions related to the value of research outputs and impact, informed by the kinds of personal and professional bias, anonymity protects against.

2. Methods

2.1 Participants

Personal contacts were used as a first point of contact and then snowballed in order to sample departmental Research Directors and REF2021 Leads, referred to collectively as ‘REF-Lead(s)’, from UK research-intensive universities. All participants were REF-Leads which meant that they had significant responsibilities for the UoA and Higher Education Institution (HEI) preparations towards their submissions to the REF2021. Care was taken to ensure representation of STEM and SSH fields, as well as representation of the UoA and Main Panel (1–4) evaluated as part of REF2021. A total of 45 individuals were invited, with n = 21 participating in the interview.

2.2 Interviews

Interviews were conducted via Zoom by Batalla, during March–June 2021 and immediately following the REF2021 submission deadline (April 2021). The time period allowed participants to reflect on their REF2021 preparation on the organizational level and emotional responses on the individual level simultaneously. This timing also reduced the risk of recall bias when reflecting on their emotional orientations to REF2021 preparations.

Interviews lasted between 40–60 min and were recorded and transcribed for analysis using the Zoom transcription platform. Transcriptions were checked for coherence against recordings by XX prior to analysis.

To ensure the interviewees’ views about the emotive interaction involved in preparations for REF2021 were not influenced by the interview discussion, as well as to characterize participant baseline emotive reaction towards REF2021, the interview was opened with a broad question regarding the individual’s professional and personal feelings of the REF and its broader civil purpose in UK’s HEI research culture (e.g. ‘As an academic, how do you personally and professionally feel about the REF?’). Following on, the interview schedule incorporated a number of themes each comprising of one, main, overarching question, followed by a series of prompts for further investigation. Importantly, the semi-structured nature of the interview schedule allowed the interview to flow as a natural discussion, rather than the interviewer introducing new concepts which could inadvertently prompt and bias a response. In this way, the interview was led by participants, and cues about the ordering and structure of the interview were taken from the interviewee. The prompts were thus used to keep the interviewee on topic, but also to allow interviewees to explore the themes mentioned by them in the interview in more depth. This participant-led approach maximized the strength of the qualitative approach adopted in this study, while also justified the analysis approach outlined below.

2.3 Interview analysis

Analysis of interview data used an inductive approach inspired by grounded theory. Such approaches use an exploratory style methodology, allowing concepts and ideas to emerge from the data (Charmaz 2006) and therefore reflect the experiences of the participants. To aid the interpretive and inductive approach to coding the data, NVivo software was used and a reflexive log, employed by the interviewer directly after conducting and/or listening to the interview recording, maintained to ensure continuity of coding across interview transcripts. This allowed the interviewer to reflect and note different themes for analysis, as well as draw parallels between interviewees as the interviews progressed.

2.4 Positionality of authors relative to participant experiences

Of the authorship team, two (Watermeyer and Derrick) had acted as REF-Leads for their own UoA and universities and significantly engaging in REF2021 preparations as reviewers for other institutions. While this experience was used as a valuable starting point for participant recruitment, as well as to the conceptualization of the approach adopted in this study, there was a parallel risk that their proximity to the experiences and emotions of the participants may have been too close to ensure sufficient distance between the experiences expressed by participants during the interviews, and the ones experienced personally. For this reason, interviews were conducted by Batalla.

We now turn to presentation of findings from our analysis which span three inter-related themes from the interviews that speak to the affective dimensions of internal departmental REF preparations: positionality tensions for REF-Leads as quasi-managers; navigating the emotional turbulence of affective auditing; and strategies for managing the impact of affective auditing.

3. Results

3.1 Positionality tensions for REF-Leads

Interviewees reported being appointed to the REF-Lead role as a consequence of previous formal involvement in REF processes, for instance as a REF Panel Member/Advisor, or as individuals whose research had been submitted to previous evaluation exercises (be that the REF2014, or its forerunner the Research Assessment Exercise (RAE)). For many, the REF-Lead role also occurred as an extension to they’re being departmental Directors of Research. Accordingly, REF-Leads are distinguished for their wealth of academic capital, which makes the role ‘easier for [them] than other people’ and that by merit of experience they are ‘in the best position to do it’:

I think I took it on because I recognised, I was in the best position to do it……Because I've been in the department for decades and because I was also lead environmental impact for the 2014 exercise, and I was slightly involved in the 2008 RAE. And I was quite experienced in how to do it, so it was easier for me to do it, than other people.

For others we consulted, becoming a REF-Lead was strategically motivated and calculated as an accelerator of career development, building upon an existing track-record of research achievement, which itself was unanimously accepted as prerequisite to the credibility of their leadership. For others still, it represents a rite of passage and professional obligation that coincides with a particular career point and threshold accumulation of capital. As one interviewee explained, assuming the role of REF-lead is inevitable once an academic’s ‘time has come’:

I think all of the leadership roles and the university, a lot of leadership roles in a university are filled by people who feel that somebody has to do it, and maybe their time is to do it now.

The hierarchical nature of REF and the kinds of capital necessary for the leadership of its institutional processes was also reflected upon by one of our interviewees who spoke of his promotion, between leading the 2014 and 2021 iterations of the REF, as an enabler of REF leadership:

But it was easier for people to respect what I was doing when I was called Professor rather not I mean I’m not sure how rational any of actually is, but it's certainly felt easier for me once I became Professor rather than reader.

However, while our interviewees discussed the importance of academic rank to the credibility of their claims to leadership, they expressed how their authority as leaders was undermined by a dearth of managerial authority connected with the role, which was perceived to be systemically scant. A dearth of managerial power afforded to them, was considered a major hurdle to their achieving staff-compliance with REF processes and, therefore, better assuring a competitive REF submission. Interviewees, were especially vocal in articulating their frustration related to a paucity of managerial power inhibiting the implementation of institutional policies at departmental levels, and in managing uncooperative researchers and/or those non-compliant with REF processes:

When I was leading [REF preparation] I had people helping me, but I was never in a management role at all.

… you're dealing with some colleagues who you feel are letting down everyone in terms of just locking themselves away and pursuing their own interests and just not giving a damn how their profile and their lack of engagement impacts on everyone else

REF-Leads tended thus to be portrayed by our interviewees as weak policy enactors, in so much as they are seen to benefit from access to high-level university management decision-making and strategic-planning and yet are deprived of managerial authority to exploit their social capital in enforcing REF policy at departmental levels. They are leaders bereft of the tools with which to enforce the actions of their leadership, and yet remain handmaidens to the REF. This dichotomy sets the scene for the emotional de-construction of REF leadership and an identity struggle that is heightened at the time of the REF.

REF-Leads are also represented as having to balance the emotional responsibility of acting both as an evaluator and themselves having their research evaluated in the REF submission preparation process. This displaced-positionality is reported as affecting the way that they manage the REF submission process and how they manage emotional interactions with colleagues connected with their own performance and REF-ability. Participation in the REF preparation process both as a REF-Lead and a REF-able researcher also seems to influence value determinations of their colleagues’ work. The appointment of the REF-Lead also sets the UoA’s threshold for ‘excellence’ in being both a role model within the department, and REF literate, even expert.

In addition to the dual-agency reported of REF-Leads as evaluating and being evaluated, interviewees articulated their sense of personal responsibility for the outcome of REF submissions and their professional vulnerability in the context of a potentially non-competitive return. Interviewees therefore internalized future REF outcomes as a reflection of their own professional performance as well as personal justification towards assuming the REF-Lead role beyond the formal REF submission:

If you're leading a unit of assessment submission, you know the buck stops with you. And you know when it all comes out, and I have anxiety about this, to this day, I mean it's been submitted, but there's going to be a result and when the result is revealed if it's poor for us there's no way that it could not reflect upon me and I'll just have to carry on.

In its basic form, the REF role instigates a number of personal anxieties and professional challenges which intensify in the run up to the REF submission deadline. We now explore how these are internalized and how REF-Leads practice resilience in preparing REF submissions.

3.2 Navigating the emotional turbulence of affective auditing

The REF-Lead role demands role-holder support while engaging the key tasks underpinning the development of a successful REF submission. In this section, we explore how the various interactions between the role-holder including both university management, and their colleagues at the departmental level intensifies emotional affect and ‘_…a conflict between the interests of the university, what the university demanded and ethical understanding of how to behave [as a colleague]’.

While the challenges of boundary-crossing highly stratified fields of activity within the university as a prestige organization are well established (cf. Watermeyer and Rowe, 2022), here we focus on how REF-Leads, like other so-called carriers of ‘middle management’ roles in universities, navigate the variety of tensions that the REF places on them. This is particularly significant given, as we have already described, the lack of managerial control by REF-Leads in navigating the repercussion of non-compliance by staff. This lack of power in necessary interactions between the REF-Lead and staff regarding their performance and subsequent contribution to the REF submission, illustrates characteristics of affective auditing but also reflects how a lack of trust, integrity and credibility of REF-Leads exacerbates the affect of the wider process.

Interviewees discussed the occupational tension of the REF-Lead role requiring representation of staff interests concurrent to their implementing institutional policies within departments. They described facing attitude-change from among their departmental colleagues caused by their privileged access to performance related data applied in selecting representations within performative artefacts like the REF’s research environment statement, and research outputs adjudged to satisfy evaluation criteria yielding return of QR² funds. Role organizational repositioning and perception change, from formative and developmental in the early years, to audit intensive in the later years of the REF cycle, is reported to significantly affect the identity and position of the REF-Lead within their departmental communities.

There are a number of strategies deployed by REF-Leads in preparing REF submissions, incentivizing participation of colleagues within REF processes while buffering them from centralized forms of command and control. One reported strategy is to protect or ‘shield’ colleagues and their research by selling REF preparation as a developmental opportunity. Interviewees, however acknowledged the potential subterfuge of altruistic claims of capacity building, associated with institutional REF processes, that are pursued less as a catalyst of equal participation and more as an instrument of enhanced performance management for low REF achievers. Relations between REF-Leads and departmental colleagues were consequently described as at risk, with vulnerabilities concerning individual performance and REF compliance, spilling over into emotive confrontations, undermining collegiality and reciprocity, and the standing of REF-Leads within departments. We find interactions involving REF-Leads and departmental staff repeatedly portrayed as ‘unpleasant conversations’:

You know it's not the bit of my job that I liked at all sitting a colleague down early on in the REF process and saying, “Actually, you know what? You're not where we want you to be and what can we do to support you to get where you want to be, and we want to help you to get where you want to be”. You know that those were unpleasant conversations, which were not much fun for the people involved and they certainly weren't much fun, for me. I did not enjoy them.

Justification for having to have such unpleasant conversations was felt to further erode with REF-Leads denied any sort of managerial authority necessary to ensuring staff-compliance with evaluative decisions, rendering them powerless to guarantee a competitive REF submission. A lack of managerial control is reflected most acutely by our REF-Leads sense of hopelessness in fixing the ‘tail of our distribution’:

There are one or two colleagues who, if we had a choice about not putting them in, we would not put them in but, given that we didn't have any choice about it and we were not going to be changing contracts, because that's not where the institution is going. Then you know we just have to live with the fact okay we'll have one or two people they'll be the tail on our distribution and so be it.

Interviewees also painfully described the emotional toll associated with explaining to colleagues that their research contribution(s), further to internal deliberations, was/were not REF-able³—judgements which might, moreover, conflict with their own personal assessment of a colleague’s research value. One interviewee described the challenge for REF-Leads in having to disassociate themselves from the impact of assessment decisions on colleagues related to their self-efficacy and sense of self-worth as researchers and in remaining necessarily dispassionate in selection decisions for departmental REF submissions:

… We might have to explain to a few colleagues that, “You know that book you're really proud of … so it's not the right kind of thing for the sort of exercise that REF is. Sorry, we liked your book it’s lovely but it's not going to perform well in the REF. Sorry we're not putting it in”. - there's a handful of things like that.

Having to deliver such evaluative verdicts does not leave REF-Leads unaffected. Instead, they must contend with the emotional response of colleagues and qualify evaluative decisions as judgements of REF alignment rather than individual research value. Of course, a decision to include or exclude outputs of any one staff member, is recognized for its impact on the psychological disposition of individual and collectives of academics and potential for upset, even humiliation: ‘I think the vast majority of staff have a very emotional response _…  on a spectrum, from anxiety to terror. Everyone is anxious.’ Despite the emotiveness of conversations focused on assessment of research outputs involving REF-Leads and departmental colleagues, and an inclination to shield staff from potentially morale damaging verdicts, many interviewees were unreserved in stating that in the REF context ‘emotions were just not relevant’. For REF-Leads, highly personal encounters would need to be entirely depersonalized, while recognizing the long-term repercussions of assessment decisions: ‘Some people are damaged by that experience for several years’.

Managing the REF-related-anxieties of colleagues was reported to amplify in the lead-up to formal institutional submission and was complicated by role reprioritizing and an increased accent on the managerial over the collegial contribution of REF-Leads. The affect here was that managing the anxieties experienced by colleagues, despite the REF pursuing a decoupling of individuals from research achievements, was akin to managing the wider wellbeing of individuals related to their performance. As one participant explained, despite REF rules excluding measurements of individual performance—and this being explicitly communicated to staff—feelings of being undervalued were explicitly communicated to REF-Leads during the process of preparing REF submissions:

… people would come back to me with anxiety that they would be undervalued, and it would come back to haunt them if they weren't valued in the REF or if they didn't have impact or something like that.

Watermeyer and Tomlinson (2022) have discussed the haunting affect of the REF and the dispossession of academics’ research identities as a consequence. In our interviews with REF-Leads, we find similar evidence of the distress caused by internal REF evaluation as a technology of control that torments and ties research identities to (monetarized) and ultimately, highly subjective interpretations of excellence, exercised by a research elite. Notwithstanding, the conspicuousness of how REF evaluations contribute to value-distortions of research(ers), REF-Leads are obligated to, at least outwardly ‘buy into’ REF evaluation decisions in order to uphold the integrity of departmental submissions, and concomitantly the efficacy of their leadership role, regardless of how ambivalent and conflicted they may be, especially in respect of sanctioning depositions of research excellence that may be factually loose and creative.

3.3 Strategies for managing the impact of affective auditing

Some of our REF-Leads attempted to mediate the ‘Negative, negative reactions or responses’ of staff during the course of departmental REF preparations, via a process of distributed leadership and the organization of evaluative expertise into committee structures (Meyer and Rowan 1977). REF committees were described as typically involving a select number of senior (REF literate) colleagues (usually but not exclusively involving the HoD and others with substantial research track-records and related academic capital), whose participation, it was felt, reinforced the authority of the REF-Lead; a case not only of weight by numbers but REF adjudications linked to the sanctioning power of line-managers. In addition, we found the moral support offered to REF-Leads through the shared responsibility of a committee structure, was rationalized by interviewees as a buffer to them being overly exposed and held singularly accountable for both the experience of developing REF submissions and their subsequent outcome: ‘In terms of moral support they very much help _…  it wasn't just me doing that it was a group of us’.

Another way with which REF-Leads are seen to alleviate the emotional baggage of REF preparations, relates to the REF Environment statement, a 15% component of the overall REF assessment and a narrative about how the submitting unit supports a research environment that is ‘vibrant’ and ‘sustainable’ (REF2021). Our interviewees signposted how this performative artefact provides an opportunity to win-over departmental colleagues, by albeit limited, consultation within its sense-making and overall crafting:

The Research Committee of the School was involved in drafting and then the whole School got to have some discussion on it so that they could feel included, involved and theoretically “buy into” all claims in our environment statement.

In such a way, research staff beyond senior department members involved in the review of research outputs and impacts, are included or rather appropriated into REF processes, even if their actual participation is highly limited. This step is important in terms both of further collectivizing responsibility for the REF—and thus insulating REF-Leads from full accountability—but also in signalling a transparent and inclusive process of selection, even if such selection is reserved for the least valuable component of evaluation.

REF-Leads self-preservation was also in interviewees’ accounts found to be practiced via emotional detachment and ‘shutting themselves away’ from the REF’s affective aspects. For some, denial of the emotional intensity of the REF is rationalized as “the only way I can be to sort of almost protect myself’. REF-Leads appear thus forced into an emotional self-isolation and estrangement from departmental colleagues, who become objects of evaluation, less persons of shared identity. They may accordingly make managerial gains in executing their REF duties, but at the cost of dehumanizing their colleagues and depriving the emotional intelligence that underpins effective leadership.

Finally, we cannot neglect the role of the COVID-19 pandemic on institutional REF preparations and specifically the ‘contribution’ of social distancing and work-from-home measures enforced in 2020 that provided REF-Leads with a hiatus from the emotional toll of organizing departmental submissions. Some participants described this as a moment diffusing tension between the REF-Leads, and their departmental colleagues; ‘I guess in a way that having this sort of breather of the last year or so and not seeing people face to face has really helped’. The hiatus from these emotional affects, therefore enabled them to facilitate and complete competitive REF submissions ahead of the deadline, which leads us to ponder the role of digitalization and/or remote forms of departmental as facilitative of REF preparation in future.

4. Discussion

Our study elucidates the affective dimensions of research evaluation on academic lives during the course of REF preparation activities within UK universities, in so doing, complementing other studies that report the significant influence of metrics and performance assessment on research communities (cf. de Rijcke et al. 2016; Söderlind and Geschwind 2019). Our interviewees describe institutional REF preparation as a series of emotional interactions involving both departmental colleagues and senior university leaders that affect research identities and departmental research communities, often in harmful ways. In this study, the needle of an emotional gauge failed to budge beyond a negative register with positive emotional associations of REF leadership conspicuously absent from our participants’ accounts, starkly revealing, therefore, consensus of REF preparations as largely unfavourable experiences for departmental communities dominated by feelings of anxiety, fear, shame, (self)disgust, and hypocrisy. These emotional contours are studied for their influence over the formation of academics’ research identity and associated value systems in reference to research evaluation, and the constitution of performance-based stratification and research hierarchies.

There are a variety of deleterious affects experienced by REF-Leads within departments, and these affects increase in intensity and diversity in the lead-up to the REF submission deadline. And while they may experiment with a variety of resilience-building techniques, increased performance pressures in the lead-up to an institution’s formal REF submission places REF-Leads in a position that isolates them from their department, as they assume the persona representative of senior leadership strategies and, by example, the neoliberal intentions of UK universities.

REF-Leads consequently occupy an antagonistic, potentially irreconcilable position in being simultaneously powerful and powerless protagonists; they suffer the differentiation and displacement of cruel optimism; ‘compromised conditions of possibility’ (Berlant 2010: 94). They are powerful in so much as the performance evaluation they oversee has huge bearing on the professional and personal wellbeing of their academic colleagues and the emotional health of the departments in which they work not only as quasi-managers but academic researchers, themselves subject to the REF’s gaze. They are powerless in so much as their appropriation into the machinery of research evaluation is as a cog, much less a lever capable of, in any meaningful and/or substantive way, influencing or disrupting the experience and affect of the REF. They are consequently controlled by the REF far more than they ever might control it and are thus vulnerable to its emotional manipulations.

Much of their vulnerability also stems from role conflict, and the specific challenge of balancing collegiality with managerial pragmatism; identity hybridity and dissonance and/or an ‘identity schism’ (Winter 2009) that stymies efforts for reconciling their dual persona as academic researchers and academic managers; and of social capital diminished by the absence of authority to legislate policy. So too, do we detect aspects of guilt from collusion with a technology of governance REF-Leads intimate is inherently flawed and inequitable. They are made further vulnerable by being internally conspicuous as policy agents sandwiched between the potential upset of departmental colleagues and the dissatisfaction of senior managers caused by their intervention.

Through these ‘passages of intensity’ (Gregg and Seigworth 2010), we find emotions exercised as cultural practice, through which bodies ‘surface’ and are ‘othered’ (Ahmed 2004, 2014). We find REF-Leads at the apotheosis of the REF as an affective economy where emotions are what bind them to those whose efforts they scrutinize and those who appraise the rigour of their scrutiny, and are also what differentiates their positionality as ‘us’ and ‘them’. As such, REF-Leads constitute ‘boundary-crossers’ (Akkerman and Bakker 2011) traversing different affective domains and thus also enacting intersecting identities. However, as the organization of the REF is cyclical, so too is the role of the REF-lead rotational and thus her/his identity and affectiveness as a formalized ‘leader’, ephemeral. Nevertheless, we would argue that such role transience no less negates the affective potency of being surfaced and othered caused by and to REF-Leads, or the potential difficulty of recovery (for all) from the emotional ‘stickiness’ (Ahmed 2014) of internal REF evaluation. Indeed, where departmental leadership of REF submissions is a focus of performance evaluation by senior institutional leaders, the threat of delivering a non-competitive REF-return consequently places REF-Leads at permanent risk of reputational, and thus career damage. They may end up both fearing and wearing the scars of emotional trauma wrought by the REF as a dehumanizing process that degrades their contribution, and that of their academic flock, to wealth creation via QR. Yet, it is this very ‘accumulation of affective value [that] shapes the surfaces of bodies and worlds’ (Ahmed 2004: 121) tied to the circulation and exchange of emotions, which makes manifest the ‘injuries’ of the REF. The emotional trauma of the REF and the intensity of the impacts of affective auditing are, however, shown by our interviewees to vary. Early Career Researchers (ECRs) and those with lower rates of research productivity, even established and/or senior academics face greatest jeopardy—and likely exhibit greatest ‘fear’ of being ‘othered’—while those with a proclivity for game-playing (including REF-Leads), able to cast themselves in the mould of the REF—and thus with proximity to the signifier of their differentiation—occupy a competitive advantage. The ‘threat’ of the REF thus modulates with different emotional positionalities. As Ahmed (2004: 127) puts it, ‘fear works to restrict some bodies through the movement or expansion of others’.

In the end, there may be no escaping the emotional turbulence of the REF process, and the precarity faced by REF-Leads in exercising rational legitimacy at the same time as fulfilling a duty of care to their departmental colleagues. The question thus arises to whom they feel most responsible for or indeed, whether the REF exorcises such concern by incentivizing the prioritization of self-interest and career survival. By viewing the REF through the lens of affect, we view not only the emotional violence committed to all those in the path of its gaze, and the normalization of the ‘unpleasantness’ of affective auditing that unsettles research identity and kinship, but a challenge to trust and integrity among academics.

Studying the emotional dimensions of research evaluation is shown to be especially pertinent given how emotional experiences, ‘leave a mark, sometimes one that is enduring and which conditions the future disposition of the subject’ (Bericat, 2016: 493), and we would add, the future sustainability of (healthy and happy) departmental research communities. Moreover, we propose that recognizing the potency of auditing processes in deleteriously affecting academic praxis is crucial to combating the manifold forms of precarization pervasive to the UK’s aggressively competitive and performance oriented higher education system. So too do we advocate affective auditing as a route to exposing the tokenism of a policy rhetoric of social responsibility—which elevates the health and welfare of staff as an institutional priority—commonly espoused by universities, yet which is neglected, if not intentionally bypassed within institutional REF preparations. The metricized Academy appears disinterested in the health and wellbeing of those it seeks to box into evaluative categories. Nevertheless, the emotional turmoil that appears to frequent REF preparations, serves to remind that research evaluation is irrefutably a social phenomenon through which bodies are bonded and concurrently displaced, that exceeds the reduction of people to numbers and challenges the quantification of academic work. Building on the work of those who have already directed their critical gaze behind-the-scenes of the REF, (cf. Derrick 2018; Bandola-Gill 2019; Watermeyer 2019; Watermeyer and Chubb 2019; Smith et al. 2020; Watermeyer and Tomlinson 2022) we accordingly present affective auditing as a further stage in the development of a human-centric analysis of research evaluation—never more so relevant than in this milieu of ‘pandemia’—which considers it not only a technology of governance but a disruptor to technologies of the self.

5. Conclusion

This study throws light onto the affective economy of research auditing, undertaken within institutions, as an affective process shaping academics’ self-actualization, which at the same time exacerbates an identity schism and the irreconcilability of aspirational or imagined and lived versions of the self, particularly among REF-Leads as ambivalent and/or conflicted protagonists. We elucidate a palimpsest of affective orientations that play out as a frenetic dance of emotions, which despite the violence of their eruption and collision, offer no recourse nor alternative from the ‘cruel optimism’ that besieges academic life and instead mark ‘a body’s belonging to a world of encounters’ (Gregg and Seigworth 2010: 2). In fact, following Berlant, we find within affective auditing the hegemony of habituation, as an invisible force which absorbs and neutralizes a political economy of disavowal and which in turn maintains and is maintained by ‘the conditions of ordinary life in the contemporary world’ being analogous to the ‘conditions of the attrition or the wearing out of the subject’ (Berlant 2010: 97). As such, we consider micro-interactions or be that micro-aggressions instigated by the REF as catalysts of affective subjectification indelibly linked to precarization and the embeddedness of a neoliberal ethos in universities, which contributes to the accretion of academics’ ontological marooning—their sense of alienation, isolation, expendability and moral compromise. The affective consequences of research evaluation in the REF context might thus be claimed as the weaponizing of researchers against themselves and the consequent means of efficient governance.

Endnotes

1

In many UK universities, a REF threshold is instated on the basis of 1-4 grading award system. In the context of grading research outputs, only those graded 3 or 4* would be eligible for a return of QR funding, with 4* outputs receiving the highest return. Thus, the focus for REF-Leads is ensuring all eligible academics returning to a UoA are able to submit at least one 3* output and the maximum of 4* outputs.

2

QR or quality-related research funds, are government funds distributed to universities relative to their performance in the REF.

3

‘REFable’ is a colloquial term used increasingly within UK academia referring to the ability of an output or individual to be of a research quality that is deemed sufficiently competitive to be submitted to the next iteration of the REF.

Ethics

Ethical clearance for this study was received from The University of Bristol Research Ethics Committee. Participants provided a signed consent form prior to the interview taking place, and verbal consent was also obtained during the interview and is reflected in the transcript. No names or identifying details (e.g. University) were used and information on gender or UoA was used primarily for the sake of comparison and validity checking during the analysis process, and not used for dissemination or publication.

Conflict of interest statement. There is no conflict of interest to declare.