Assessing research excellence: Evaluating the Research Excellence Framework

Abstract

Performance-based research funding systems have been extensively used around the globe to allocate funds across higher education institutes (HEIs), which led to an increased amount of literature examining their use. The UK’s Research Excellence Framework (REF) uses a peer-review process to evaluate the research environment, research outputs and non-academic impact of research produced by HEIs to produce a more accountable distribution of public funds. However, carrying out such a research evaluation is costly. Given the cost and that it is suggested that the evaluation of each component is subject to bias and has received other criticisms, this article uses correlation and principal component analysis to evaluate REF’s usefulness as a composite evaluation index. As the three elements of the evaluation—environment, impact and output—are highly and positively correlated, the effect of the removal of an element from the evaluation leads to relatively small shifts in the allocation of funds and in the rankings of HEIs. As a result, future evaluations may consider the removal of some elements of the REF or reconsider a new way of evaluating different elements to capture organizational achievement rather than individual achievements.

1. Introduction

Performance-based research funding systems (PRFS) have multiplied since the United Kingdom introduced the first ‘Research Selectivity Exercise’ in 1986. Thirty years on from this first exercise, Jonkers and Zacharewicz (2016) reported that 17 of the EU28 countries had some form of PRFS, and this had increased to 18 by 2019 (Zacharewicz et al. 2019).

A widely used definition of what constitutes a PRFS is that they must meet the following criteria (Hicks 2012):

• Research must be evaluated, not the quality of teaching and degree programmes;

• The evaluation must be ex post, and must not be an ex ante evaluation of a research or project proposal;

• The output(s) of research must be evaluated;

• The distribution of funding from Government must depend upon the evaluation results;

• The system must be national.

Within these relatively narrow boundaries, there is significant variation between both what is assessed in different PRFS, and how the assessment is made. With regards to ‘what’, some focus almost exclusively on research outputs, predominantly journal articles, whereas others, notably the UK’s Research Excellence Framework (REF), assess other aspects of research such as the impact of research and the research environment. With regards to ‘how’, some PRFS use exclusively or predominantly metrics such as citations whereas others use expert peer review, and others still a mix of both methods (Zacharewicz et al. 2019).¹

This article focuses on UK’s REF, which originated in the very first PRFS, the Research Selectivity Exercise in 1986. This was followed by a second exercise in 1989 and a series of Research Assessment Exercises (RAEs) in the 1990s and 2000s. Each RAE represented a relatively gentle evolution from the previous one, but there was arguably more of a revolution than evolution between the last RAE in 2008 and the first REF in 2014 (REF 2014), with the introduction of the assessment of research impact into the assessment framework (see e.g., Gilroy and McNamara 2009; Shattock 2012; Marques et al. 2017 for a detailed discussion on the evolution of RAEs in the UK). Three elements of research, namely research outputs, the non-academic impact of research and the research environment, were evaluated in the REF 2014 exercise. Research outputs (e.g., journal articles, books and research-based artistic works) were evaluated in terms of their ‘originality, significance and rigour’. The assessment of the non-academic impact of research is based on the submission of impact case studies that describe the details of the ‘reach and significance’ of impacts on the economy, society and/or culture, that were underpinned by excellent research. The research environment consisted of both data relating to the environment and a narrative environment statement. The environment data consisted of the number of postgraduate research degree completions and total research income generated by the submitting unit. The research environment statement provided information on the research undertaken, the staffing strategy, infrastructure and facilities, staff development activities, and research collaborations and contribution to the discipline. The quality of the research environment was assessed in terms of its ‘vitality and sustainability’ based on the environment data and narrative environment statements (see REF 2012 for further details).

There has been criticism of several aspects of the assessment of research excellence in the REF, including the cost of preparation and evaluation of the REF, the potential lack of objectivity in assessing them and the effect of the quasi-arbitrary or opaque value judgements on the allocation of quality-related research (QR) funding (see Section 2 for the details). Furthermore, the use of multiple criteria, which is the case for the REF (i.e., environment, impact and outputs), in assessing university performance has been long criticized (see e.g., Saisana, d’Hombres and Saltelli 2011; Pinar, Milla and Stengos 2019). These multidimensional indices are risky as some of the index components have been considered redundant (McGillivray 1991; McGillivray and White 1993). For instance, McGillivray (1991), McGillivray and White (1993) and Bérenger and Verdier-Chouchane (2007) use correlation analysis to examine the redundancy of different components of well-being when the indices are constructed. The main argument of these papers is that if the index components are highly and positively correlated, then the inclusion of additional dimensions to the index does not add new information to that provided by any of the other components. Furthermore, Nardo et al. (2008) also point out that obtaining a composite index with the highly correlated components leads to a double weighting of the same information and so overweighting of the information captured by these components. Therefore, this literature argues that excluding any component from the evaluation does not lead to loss of information if the evaluation elements are highly and positively correlated. For instance, by using correlation analysis, Cahill (2005) showed that excluding any component from a composite index produces rankings and achievements similar to the composite index. To overcome these drawbacks, principal components analysis (PCA) has been used to obtain indices (see e.g., McGillivray 2005; Khatun 2009; Nguefack‐Tsague, Klasen and Zucchini 2011 for the use of PCA for well-being indices, and see Tijssen, Yegros-Yegros and Winnink 2016 and Robinson-Garcia et al. 2019 for the use of PCA for university rankings). The PCA transforms the correlated variables into a new set of uncorrelated variables using a covariance matrix, which explains most of the variation in the existing components (Nardo et al. 2008).

This article will contribute to the literature by examining the redundancy of the three components of the REF by using the correlation analysis between them to examine the relevance of each component for the evaluation. If the three elements of the REF are highly and positively correlated, then excluding one component from the analysis will not result in major changes in the overall assessment of universities and funding allocated to them. This article will examine whether this would be the case. Furthermore, we will also carry out PCA to obtain weights that would produce an index that explains most of the variation in the three elements of the REF while obtaining an overall assessment of higher education institutes (HEIs) and distributing funding across them.

The remainder of this article is structured as follows. In Section 2, we will provide details on how the UK’s REF operates, identify the literature on the REF exercise and outline the hypotheses of the article. In Section 3, we provide the detailed data used in this article and examine the correlation between the environment, impact and output scores. In this section, we also provide the details of the QR funding formula used to allocate the funding and demonstrate the correlation between the funding distribution in the environment, impact and output pots. We also will carry out PCA by using the achievement scores and funding distributed in each element in this section. Finally, in this section, we provide an alternative approach to the calculation of overall REF scores and the distribution of QR funding based on the hypotheses of the article. Section 4 will consider the effect on the distribution of QR funding for English universities² and their rankings when each element is removed from the calculation one at a time and PCA weights are used. Finally, Section 5 will identify conclusions of our analyses and the implications for how future REF assessment exercises might be structured.

2. Research Excellence Framework and related literature

Research assessment exercises have existed in the UK since the first Research Selectivity Exercise was undertaken in 1986. A subsequent exercise was held in 1989, which was followed by RAEs in 1996, 2001 and 2008. Each HEI’s submission to the 1986 exercise comprised a research statement in one or more of 37 subject areas, together with five research outputs per area in which a submission was made (see e.g., Hinze et al. 2019). The complexity of the submissions has increased from that first exercise, and in 2014 the requirement to submit case studies and a narrative template to allow for the assessment of research impact was included for the first time, and the exercise was renamed to the REF.

The REF 2014 ‘Assessment Framework and Guidance on Submissions’ (REF 2011) indicated that a submission’s research environment would be assessed according to its ‘vitality and sustainability’, using the same five-point (4* down to unclassified) scale as for the other elements of the exercise.³

Following the 2014 REF exercise, there have been many criticisms of REF. For instance, the effects of the introduction of impact as an element of the UK’s research assessment methodology has itself been the subject of many papers and reports in which many of the issues and challenges it has brought have been discussed (see e.g., Smith and Ward, House 2011; Penfield et al. 2014; Manville et al. 2015; Watermeyer 2016; Pinar and Unlu 2020a). Manville et al. (2015) and Watermeyer (2016) show that academics in some fields were concerned about how their research focus would be affected by the impact agenda by forcing them to produce more ‘impactful’ research than carrying out their own research agenda. On the other hand, Manville et al. (2015) demonstrate that there have been problems with the peer reviewing of the impact case studies where reviewer panels struggled to distinguish between 2-star and 3-star and, most importantly, between 3-star and 4-star. Furthermore, Pinar and Unlu (2020a) demonstrate that the inclusion of the impact agenda in REF 2014 increased the research income gap across HEIs. Similarly, the literature identifies some serious concerns with the assessment of the research environment (Taylor 2011; Wilsdon et al. 2015; Thorpe et al. 2018a,b). Taylor (2011) considered the use of metrics to assess the research environment, and found evidence of bias towards more research-intensive universities in the assessment of research environment in the 2008 RAE (see Pinar and Unlu 2020b for similar findings for the REF 2014). In particular, he argued that the judgement of assessors may have an implicit bias and be influenced by the ‘halo effect’, where assessors allocate relatively higher scores to departments with long-standing records of high-quality research, and showed that members of Russell Group universities benefited from a ‘halo effect’, after accounting for various important quantitative factors. Wilsdon et al. (2015) wrote in a report for the Higher Education Funding Council for England (HEFCE), which ran the REF on behalf of the four countries of the UK, in which those who had reviewed the narrative research environment statements in REF 2014 as members of the panels of experts expressed concerns ‘that the narrative elements were hard to assess, with difficulties in separating quality in research environment from quality in writing about it.’ Thorpe et al. (2018a,b) examined environment statements submitted to REF 2014, and their work indicates that the scores given to the overall research environment were influenced by the language used in the narrative statements, and whether or not the submitting university was represented amongst those experts who reviewed the statements. Finally, a similar peer-review bias has been identified in the evaluation of research outputs (see e.g., Taylor 2011). Overall, there have been criticisms about the evaluation biases in each element of the REF exercise.

Another criticism of the REF 2014 exercise has been that of the cost. HEFCE commissioned a review of it (Farla and Simmonds 2015) which estimated the cost of the exercise to be £246 million (Farla and Simmonds 2015, 6), and the cost of preparing the REF submissions was £212 million. It can be estimated that roughly £19–27 million was spent preparing the research environment statements,⁴ and £55 million was spent in preparation of impact case studies, and the remainder cost of preparation may be associated with the output submission. Overall, the cost of preparing each element was significant. Since there is a good agreement between bibliometric factors and peer review assessments (Bertocchi et al. 2015; Pidd and Broadbent 2015), it has been argued that cost of evaluating outputs could be decreased with the use of bibliometric information (see e.g., De Boer et al. 2015; Geuna and Piolatto 2016). Furthermore, Pinar and Unlu (2020b) found that the use of ‘environment data’ alone could minimize the cost of preparation of the environment part of the assessment as the environment data (i.e., income generated by units, number of staff and postgraduate degree completions) explains a good percentage of the variation between HEIs in REF environment scores.

Because of these criticisms, together with Kelly (2016) and Pinar (2020)’s works which show that a key outcome of the REF, which is to distribute ca. £1bn per annum of QR funding, is dependent upon somewhat arbitrary or opaque value judgements (e.g., the relative importance of world-leading research compared to internationally excellent research and the relative cost of undertaking research in different disciplines). In this article, we will contribute to the existing literature by using correlation analysis to examine the redundancy of each research element, and also use PCA to obtain weights for each element that overcome high correlation between three elements but explain most of the variation in achievements and funding distribution in each element.

The three components of the REF are highly and positively correlated (see next section for correlation analysis), and a high and positive correlation amongst the three components would suggest that removal of one component from the REF would have only a small effect on the QR funding distribution and overall performance rankings based on the redundancy literature (e.g., McGillivray 1991; McGillivray and White 1993; Bérenger and Verdier-Chouchane 2007). Therefore, based on the arguments put forward in the redundancy literature, we set the hypotheses of this article as follows:

Hypothesis 1: Exclusion of one of the REF elements from the distribution of the mainstream QR funding would lead to relatively small shifts in the allocation of funds if three components of the REF elements are positively and highly correlated.

Hypothesis 2: Exclusion of one of the REF elements from the calculation of overall REF grade point averages (GPAs) obtained by HEIs would result in relatively small shifts in the rankings of HEIs when REF elements are positively and highly correlated.

Hypothesis 3: Overall REF GPAs and allocation of funding with the PCA weights given to each element of REF would result in small shifts in rankings and funding allocation when three components of the REF are highly and positively correlated.

3. Methodology

In this section, we will provide the details of the data sources for the REF results and QR funding allocation based on the REF results. We will also discuss the alternative ways of obtaining overall REF scores and QR funding allocation.

3.1 REF results data

In REF 2014, each participating UK institution submitted in one or more disciplinary areas, known as ‘units of assessment’ (UOAs). Each submission comprised three elements:

• A number of research outputs. The expected number of research outputs submitted by each UOA was four times the full-time equivalent (FTE) staff included in that submission, unless one or more staff members was allowed a reduction in outputs. Each FTE staff member was expected to submit four research outputs, but reductions in outputs were allowed for staff members who had individual circumstances which included that they were early career researchers, had taken maternity, paternity or adoption leave during the assessment period, or had had health problems.

• A number of case studies demonstrating the impact of research undertaken within that UOA, and a narrative ‘impact template’ which included a description of the UOA’s approach to generating impact from its research. Each case study was a maximum of four pages and the rules stipulated that the number of case studies required depended upon the number of FTEs submitted in the UOA, as was the length of the impact template. Ninety-five per cent of submissions by English universities comprised between two and seven case studies and narratives that were three or four pages long.⁵

• Information about the research environment, which comprised a narrative ‘environment statement’ describing the research environment, together with data on research income and PhD completions. As with the impact narrative the length of the environment statement was dependent upon the number of FTEs submitted, with 95% of submission from English universities comprising narratives which were between 7 and 12 pages long.

After the submission of UOAs, each individual component in these elements (e.g., a research output, an impact case study) was given a score on a five-point ‘star’ scale, namely 4* (world-leading), 3* (internationally excellent), 2* (internationally recognized), 1* (nationally recognized) and unclassified (for elements which were below the 1* standard) by the peer-reviewers. From the scores for each individual component in each element, a profile for each element was obtained and this was the information which was released by HEFCE. This profile for each element, obtained from REF (2014) gives the percentage of the research in each element (i.e., research outputs, environment and impact) that were rated as 4*, 3*, 2*, 1* or unclassified. Finally, an overall research profile of the UOA is calculated where each element’s score was weighted 65:20:15 for outputs: impact: environment.

To test whether the quality of the research environment, impact and outputs are correlated, we obtain each individual submissions’ weighted average environment, impact and output scores.⁶  Table  1 provides a correlation matrix between GPA scores of different elements. This table shows that GPA scores are positively and significantly correlated with each other at the 1% level. Table  2 shows the results of PCA of the three elements when GPA scores in each element are used. The first principal component accounts for approximately 79.0% of the variation in three elements. In comparison, the first two principal components account for approximately 92.5% of the variation in three elements. Clearly, the first principal component contains most of the statistical information embedded in the three elements. Second, the first principal component results in roughly similar eigenvectors, suggesting that the overall GPA scores could be obtained using roughly equal weights given to each element when the eigenvectors are normalized to sum the weights to 1.

Table 1.

Correlation matrix between different element GPAs

	Environment	Impact	Output
Environment	1
Impact	0.73*	1
Output	0.72*	0.59*	1

Note: Asterisk (*) represents a significance level at the 1% level.

Table 2.

Results of PCA of the three elements using GPA scores

	Eigenvalue	Proportion of variance explained	Cumulative proportion of variance explained
Component 1	2.34097	0.7870	0.7870
Component 2	0.41443	0.1381	0.9251
Component 3	0.22461	0.0749	1.0000

	Principal component 1	Normalized weights
Environment	0.602	0.348
Impact	0.566	0.327
Output	0.563	0.325

Since all the elements are positively and significantly correlated with each other, removing one of the elements from the REF assessment or an alternative combination of the REF elements (via PCA weights) might have a little overall effect on the distribution of QR income and overall achievement.

3.2 QR funding allocation data based on REF results

Based on the REF results obtained by UOAs, Research England describes how it distributes QR funding in Research England (2019a). In brief, QR funding comprises six elements: (1) mainstream QR funding; (2) QR research degree programme supervision fund; (3) QR charity support fund; (4) QR business research element; (5) QR funding for National Research Libraries; and (6) the Global Challenge Research Fund. The mainstream QR funding is the largest, comprising approximately two-thirds of the overall QR funding, and is the element which is most directly related to an institution’s performance in REF 2014. The data for the mainstream QR funding allocations across panels, UOAs and HEIs during the 2019–20 funding period are obtained from Research England (2019b).

In calculating an institution’s mainstream QR funding, Research England follows a four-stage process:

1. The mainstream QR funding is separated into three elements, for outputs, impact and environment, with 65% of funding for outputs, 20% for impact and 15% for environment.

2. The funding for each of the three elements is distributed amongst the four ‘main subject panels’⁷ in proportion to the volume of research in each main panel which was 3* or above, weighted to reflect an assumed relative cost of research in different disciplines.

3. Within each main panel, mainstream QR funding is distributed to each UOA according to the volume of research at 3* or above and the cost weights (which reflect the relative cost of undertaking research in different disciplines), and with an additional multiplier of 4 being given to research rated as world-leading, i.e., 4* research, compared to internationally excellent, or 3*, research.

4. The mainstream QR funding for each element in each UOA is then distributed to individual HEIs according to the volume of research at 3* or above produced by that HEI, with the cost and quality weights taken into account.

Therefore, a university’s total QR mainstream funding comprises an amount for each element of outputs, impact and environment, for each UOA in which it made a submission.

Since the allocation of the mainstream QR funding in each pot (environment, impact and output) is closely related to the performance of the UOAs in each respective research element, we also found positive and significant correlation coefficients between mainstream QR funding distributed to the UOAs in the environment, impact and output pots at the 1% level (see Table  3). Similarly, when we carried out PCA analysis, we found that the first principal component accounts for approximately 97% of the variation in the components, and the first principal component results in roughly similar eigenvectors (see Table  4), suggesting that equal funding could be distributed in the environment, impact and output pots.

Table 3.

Correlation matrix between different funding pots

	Environment	Impact	Output
Environment	1
Impact	0.96*	1
Output	0.95*	0.94*	1

Note: Asterisk (*) represents a significance level at the 1% level.

Table 4.

Results of PCA of the three elements using funds distributed in each pot

	Eigenvalue	Proportion of variance explained	Cumulative proportion of variance explained
Component 1	2.90494	0.9683	0.9683
Component 2	0.05567	0.0186	0.9869
Component 3	0.03939	0.0131	1.0000

	Principal component 1	Normalized weights
Environment	0.579	0.334
Impact	0.577	0.333
Output	0.576	0.333

3.3 Alternative ways of allocating QR funding and obtaining overall REF scores

Based on the arguments in the redundancy literature, we examine the effects of excluding one element of the evaluation while distributing QR funding and calculating overall REF scores. Initially, as described in Section 3.2, the mainstream QR funding is distributed across three pots (i.e., output, environment and impact) where 65%, 20% and 15% of the mainstream QR funding is distributed based on the performances of the submissions in output, impact and environment elements in REF 2014, respectively (i.e., step 1 of the funding formula). Similarly, the overall REF scores of units and HEIs were obtained by a weighted average of the three elements where the output, impact and environment performances were weighted 65%, 20% and 15%, respectively. If one of the elements (i.e., environment, impact and output) is excluded, the weight given to it should be allocated amongst the other two elements to redistribute the QR funding and to obtain the overall REF scores, so that the weights sum to 100%. In the first scenario, we exclude the environment element and reallocate the weight of environment to output and impact in proportion to their initial weights: 65:20, which becomes 76.5% and 23.5%.⁸ For the second scenario, we exclude the impact element and reallocate the weight of impact to the environment and output in proportion to their initial weights: 15:65, which results in 18.75% and 81.25%. Finally, if we exclude the output element, then the environment and impact elements are allocated 43% and 57% weights based on their initial weight ratio of 15:20. Finally, as a fourth scenario, we rely on the results obtained with the PCA and that each element is kept in the calculation of the overall GPA and distribution of QR funding, but instead, each element is given equal weights (i.e., 33.33%).

Based on the funding formula of the mainstream QR funding allocation (see Research England 2019a, 16–9 for details on how the mainstream QR funding is allocated or Section 3.2 of this article for the steps), we follow the same steps to redistribute the mainstream QR funding across different panels, UOAs and HEIs based on the alternative scenarios. To obtain the overall REF scores of HEIs, the overall GPA of each unit is obtained by weighting the GPA of output, impact and environment elements with 65%, 20% and 15%, respectively. With the alternative scenarios, we will obtain the overall GPA of the HEIs by weighting elements with respective scenario weights as discussed above.

4. Results

4.1 Alternative way of allocating QR funding

In this subsection, we will examine the effect of the mainstream QR funding distribution to different panels, UOAs, and HEIs in England with Scenarios 1–4 compared to the official mainstream QR funding allocation. To provide an overall picture of the amount of mainstream QR funding distributed in 2019–20 funding period, Table  5 provides the amount of mainstream QR funding distributed in each of the three pots with the official REF 2014 results, and mainstream QR funding distributed with the alternative scenarios proposed in this article. During the 2019–20 funding period, a total of £1,060 million (i.e., just over a billion pounds) was distributed under the mainstream QR funding and roughly £159 million, £212 million and £689 million of mainstream QR funding are distributed in the environment, impact and output pots across the English HEIs, respectively.⁹ On the other hand, with Scenarios 1, 2 and 3, no mainstream QR funding is distributed in the environment, impact and output pots, respectively. Whereas, equal amounts of funds are distributed in each pot with Scenario 4. With Scenario 1, £249 million and £811 million of mainstream QR funding are distributed based on the REF 2014 performances in impact and output elements, indicating that an additional £37 million and £122 million are distributed in the impact and output pots compared to the official scenario, respectively. In contrast, with Scenario 2, £199 million and £862 million of mainstream QR funding are distributed in environment and output elements, indicating that additional £39 million and £72 million are distributed in the environment and output pots compared to the official scenario, respectively. In Scenario 3, £456 million and £605 million are distributed in environment and impact pots, respectively, suggesting that additional £297 million and £392 million were distributed in respective pots compared to the official scenario. Finally, with Scenario 4, an equal amount of funds (i.e., £353.5 million) are distributed in each pot where more funding is allocated in environment and impact pots and less funding is distributed in output pot.

Table 5.

Distribution of mainstream QR funding across different pots based on the REF2014 results and alternative scenarios

Pots	REF2014 (£)	REF2014 (%)	Scenario 1 (£)	Scenario 1 (%)	Scenario 2 (£)	Scenario 2 (%)	Scenario 3 (£)	Scenario 3 (%)	Scenario 4 (£)	Scenario 4 (%)
Environment	159,106,561	15	0	0	198,883,217.1	18.75	456,105,511.1	43	353,570,163.7	33.33%
Impact	212,142,100	20	249,266,965.4	23.5	0	0	604,604,979.9	57	353,570,163.7	33.33%
Output	689,461,830	65	811,443,525.6	76.5	861,827,273.9	81.25	0	0	353,570,163.7	33.33%

Table  6 provides the allocation of the mainstream QR funding to four main panels (i.e., Panel A: Medicine, Health and Life Sciences; Panel B: Physical Sciences, Engineering and Mathematics; Panel C: Social Sciences; Panel D: Arts and Humanities) with the REF 2014 results, and with alternative scenarios. This table also provides the change in the mainstream QR funding received by four main panels from the official allocation to alternative scenarios where a positive (negative) change indicates that the panel would have received more (less) funding with the alternative scenario compared to the official allocation. The results suggest that the panel B would have been allocated more funds, and panels A, C and D would have been allocated less QR funding with the alternative Scenarios 1 and 2 compared to the official allocation, suggesting that exclusion of environment and impact elements would have benefitted panel B. On the other hand, panel B (panels A, C and D) would have generated less (more) QR funding with the third and fourth scenarios (i.e., when the output element is excluded, and equal amount of funds distributed in each pot, respectively) compared to the official scenario. Overall, with the reallocation of QR funding with alternative Scenarios 1, 2, 3 and 4, only 0.34%, 0.64%, 2.29% and 1.08% of the total mainstream QR funding (i.e., £3.6 million, £6.8 million, £24.3 million and £11.5 million) would have been reallocated across the four main panels with the alternative allocation scenarios compared to the official one.

Table 6.

Allocation of the mainstream QR funding to four main panels with the alternative scenarios

Panel	REF2014 scenario (£)	Scenario 1 (£)	Scenario 2 (£)	Scenario 3 (£)	Scenario 4 (£)	Scenario 1— REF2014 (£)	Scenario 2— REF2014 (£)	Scenario 3— REF2014 (£)	Scenario 4— REF2014 (£)
Panel A	338,298,368	336,366,259	335,407,018	349,585,907	343,767,105	−1,932,109	−2,891,350	11,287,539	5,468,737
Panel B	343,928,912	347,509,212	350,766,784	319,643,542	332,432,163	3,580,300	6,837,872	−24,285,370	−11,496,749
Panel C	215,592,577	214,727,107	213,077,971	223,424,509	219,165,171	−865,470	−2,514,606	7,831,932	3,572,594
Panel D	162,890,634	162,107,914	161,458,718	168,056,533	165,346,052	−782,720	−1,431,916	5,165,899	2,455,418

Note: Panels A (Medicine, Health and Life Sciences), B (Physical Sciences, Engineering and Mathematics), C (Social Sciences) and D (Arts and Humanities) consist of the UoAs between 1 and 6, 7 and 15, 16 and 26, and 27 and 36, respectively.

Table  7 reports the official QR funding allocation and the QR funding allocation changes between alternative scenarios and official scenario in different UOAs where a positive (negative) figure suggests that the UOA received relatively more (less) QR funding with the alternative scenario compared to the official case. We find, for example, that the Computer Science and Informatics, and the Public Health, Health Services and Primary Care units would have received £2.0 million more and £1.2 million less QR funding when the environment element is excluded (Scenario 1) compared to the official scenario, respectively. On the other hand, when the impact element is excluded (Scenario 2), the Biological Sciences and Clinical Medicine units would have generated £3.0 million more and £4.3 million less than the official scenario, respectively. When the output element is excluded from the evaluation (Scenario 3), we find that the Clinical Medicine and Biological Sciences units would have generated £11.7 million more and £7.2 million less compared to the official scenario, respectively. Finally, if all three elements are weighted equally (Scenario 4), Clinical Medicine and Computer Science and Informatics units would have generated £5.1 million more and £3.5 million less than the official scenario, respectively. This evaluation clearly shows in which elements specific subjects perform better (worse) than other subject areas. Even though we observe changes in funds generated by each unit with alternative scenarios, there is a limited funding shift across units. Overall, the total amounts reallocated across different UOAs are £5.9 million, £11.5 million, £36.9 million and £17.2 million with Scenarios 1, 2, 3 and 4, which correspond to 0.55%, 1.08%, 3.48% and 1.62% of the total mainstream QR funding, respectively.

Table 7.

Allocation of mainstream QR funding across different UoAs and changes in funding allocation with alternative scenarios compared to benchmark

UoA no	UoA name	Official (£)	Scenario 1— Official (£)	Scenario 2— Official (£)	Scenario 3— Official (£)	Scenario 4— Official (£)
1	Clinical Medicine	95,523,490	−747,318	−4,338,117	11,694,917	5,080,152
2	Public Health, Health Services and Primary Care	37,955,163	−1,240,399	−1,399,718	6,207,389	3,090,516
3	Allied Health Professions, Dentistry, Nursing and Pharmacy	61,304,968	645,471	729,643	−3,233,045	−1,609,390
4	Psychology, Psychiatry and Neuroscience	64,233,254	−160,568	−444,319	1,402,483	642,362
5	Biological Sciences	61,974,301	174,671	2,961,800	−7,167,257	−2,981,049
6	Agriculture, Veterinary and Food Science	17,307,192	−603,965	−400,639	2,383,053	1,246,145
7	Earth Systems and Environmental Sciences	31,339,762	227,841	125,337	−840,259	−446,340
8	Chemistry	34,227,000	737,462	832,581	−3,691,425	−1,837,791
9	Physics	46,860,732	43,186	1,730,417	−4,044,056	−1,656,166
10	Mathematical Sciences	55,246,391	−91,900	2,103,527	−4,564,314	−1,803,537
11	Computer Science and Informatics	47,248,292	2,033,690	584,702	−6,284,448	−3,492,216
12	Aeronautical, Mechanical, Chemical and Manufacturing Engineering	31,644,046	169,460	61,093	−551,821	−302,386
13	Electrical and Electronic Engineering, Metallurgy and Materials	29,742,204	853,694	660,528	−3,582,897	−1,848,176
14	Civil and Construction Engineering	8,829,420	−43,753	165,690	−270,583	−89,026
15	General Engineering	58,791,065	−349,380	573,998	−455,569	−21,111
16	Architecture, Built Environment and Planning	16,687,239	−187,682	−15,796	493,098	287,141
17	Geography, Environmental Studies and Archaeology	34,567,204	1	−1	0	−1
18	Economics and Econometrics	13,125,223	354,761	212,924	−1,348,771	−711,336
19	Business and Management Studies	44,327,243	74,200	−420,647	776,766	279,578
20	Law	21,483,983	−538,345	152,116	965,008	641,835
21	Politics and International Studies	17,356,122	−143,310	−504,397	1,497,198	672,618
22	Social Work and Social Policy	16,751,570	62,620	−596,292	1,204,786	458,140
23	Sociology	9,426,140	129,307	−348,854	479,122	133,429
24	Anthropology and Development Studies	7,761,232	−246,938	−233,737	1,133,519	573,896
25	Education	20,703,573	−362,951	−359,592	1,702,571	858,289
26	Sport and Exercise Sciences, Leisure and Tourism	13,403,048	−7,136	−400,330	928,635	379,006
27	Area Studies	7,297,713	−49,046	−176,533	521,296	233,795
28	Modern Languages and Linguistics	18,544,871	−204,340	−13,798	529,127	309,497
29	English Language and Literature	29,817,530	−214,962	255,106	−57,097	77,305
30	History	26,345,506	85,814	283,857	−855,148	−386,026
31	Classics	6,477,415	−15,311	−118,780	307,666	131,615
32	Philosophy	8,653,123	−105,244	54,697	131,842	102,493
33	Theology and Religious Studies	5,146,244	22,924	−69,140	101,543	30,371
34	Art and Design: History, Practice and Theory	26,565,280	−412,265	−928,766	3,118,272	1,454,034
35	Music, Drama, Dance and Performing Arts	20,875,925	250,835	−556,772	656,389	148,379
36	Communication, Cultural & Media Studies, Library & Information Management	13,167,027	−141,126	−161,786	712,009	353,955

Note: A positive (negative) figure in changes columns suggests that the UOA received relatively more (less) QR funding with the respective alternative scenario compared to the official case.

Finally, we examine the effect of alternative QR funding allocations on the funding received by HEIs. Table  8 shows the five HEIs that would have generated the biggest increase (decrease) in mainstream QR funding with the alternative scenarios compared to the official allocation. The data show that the University of Leicester, University of Plymouth, University of East Anglia, University of Birmingham and the University of Surrey would have generated £745k, £552k, £550k, £522k and £464k more QR funding with the first scenario compared to the official scenario, whereas University College London, University of Cambridge, University of Oxford, University of Manchester and the University of Nottingham would have generated £3.4 million, £2.1 million, £2million, £1.5 million and £1.4 million less, respectively. On the other hand, the University of Cambridge would have generated £1.9 million more if the impact element is excluded (Scenario 2), and University College London would have generated £9.8 million and £5.6 million more if the output element is excluded (Scenario 3) and each element is weighted equally (Scenario 4), respectively. In comparison, the University of Leeds, University of Birmingham and University of Leicester would have generated £1 million, £2.4 million and £1.3 million less with Scenarios 2, 3 and 4, respectively. Overall, the total amounts reallocated across different HEIs are £15.5 million, £11.1 million, £46.7 million and £25.6 million with Scenarios 1, 2, 3 and 4, which correspond to just 1.46%, 1.05%, 4.41% and 2.42% of the total mainstream QR funding, respectively. Furthermore, only a handful of universities would have experienced a significant change in their funding allocation with alternative scenarios where 6, 3, 25 and 10 HEIs experienced a difference in their QR funding allocation of more than £1 million with Scenarios 1, 2, 3 and 4 compared to the official one, respectively (see Appendix Table A.1 for the allocation of the mainstream QR funding to the HEIs with the official case and also the difference in the allocation of QR funding between alternative scenarios and official one).

Table 8.

Five HEIs that would generate more (less) with the alternative scenarios compared to the official scenario

Panel A. HEIs with more (less) QR funding with the Scenario 1 compared to the official case
Institution	Scenario 1—Official (£)	Institution	Scenario 1—Official (£)
The University of Leicester	744,644	University College London	−3,370,285
University of Plymouth	551,610	University of Cambridge	−2,098,481
The University of East Anglia	549,917	University of Oxford	−1,952,481
The University of Birmingham	521,852	The University of Manchester	−1,537,168
The University of Surrey	463,899	University of Nottingham, The	−1,447,040

Panel B. HEIs with more (less) QR funding with the Scenario 2 compared to the official case
Institution	Scenario 2—Official (£)	Institution	Scenario 2—Official (£)
University of Cambridge	1,893,673	The University of Leeds	−1,013,835
University of Oxford	1,360,203	University of Bristol	−803,044
The University of Warwick	966,259	University College London	−714,947
University of Exeter	494,541	The University of Manchester	−652,384
The University of Birmingham	484,144	University of Durham	−649,139

Panel C. HEIs with more (less) QR funding with the Scenario 3 compared to the official case
Institution	Scenario 3—Official (£)	Institution	Scenario 3—Official (£)
University College London	9,836,506	The University of Birmingham	−2,373,126
The University of Manchester	5,229,119	The University of Leicester	−2,350,009
University of Nottingham, The	4,743,660	The University of Surrey	−2,165,688
Imperial College of Science, Technology and Medicine	3,937,198	University of Plymouth	−2,099,967
The University of Leeds	3,134,843	The University of Warwick	−2,047,730

Panel D. HEIs with more (less) QR funding with the Scenario 4 compared to the official case
Institution	Scenario 4—Official (£)	Institution	Scenario 4—Official (£)
University College London	5,553,467	The University of Leicester	−1,298,485
The University of Manchester	2,833,377	The University of Birmingham	−1,203,776
University of Nottingham, The	2,594,892	University of Plymouth	−1,107,171
Imperial College of Science, Technology and Medicine	2,180,722	The University of Surrey	−1,092,790
University of Oxford	1,583,315	Queen Mary University of London	−895,910

4.2 Ranking of HEIs

Since the REF exercise is used in the rankings of HEIs, in this subsection, we will evaluate the effect of different scenarios on the overall GPA and rankings of HEIs. Table  9 offers the Spearman’s rank correlation coefficients between GPA scores obtained with the official scenario and the GPA scores obtained with the alternative scenarios. We find that the GPA scores obtained with the alternative scenarios are highly and positively correlated with the official GPA scores at the 1% level. Even though the correlation coefficients between GPA scores of HEIs with the alternative scenarios and official one are highly and positively correlated, some HEIs would have been ranked in relatively higher (lower) positions with the alternative scenarios compared to the official scenario. Amongst 111 HEIs, just 9, 5, 22 and 5 HEIs experienced more than 10 position changes in their ranking with the Scenarios 1, 2, 3 and 4, respectively, compared to the official rankings. For instance, Guildhall School of Music & Drama would have experienced a major improvement in their ranking with the third scenario as it would have been ranked in the 53rd position when output element is excluded (i.e., Scenario 1) compared to the 89th position with the official scenario. On the other hand, London Business School would have been ranked in the 32nd position with the third scenario, but ranked 7th with the official scenario (see Appendix Table  A.2 for the GPA scores and respective rankings of HEIs with the official case and Scenarios 1, 2, 3 and 4). However, with very few exceptions, it can be seen that the difference between the rankings in the alternative scenarios compared with the official rankings is relatively small.

Table 9.

Spearman’s rank correlation coefficients between official and alternative scenario GPAs

	GPA— Scenario 1	GPA— Scenario 2	GPA— Scenario 2	GPA— Scenario 2
GPA—REF2014	0.9900*	0.9885*	0.9596*	0.9870*

Note: Asterisk (*) represents significance level at the 1% level.

5. Conclusions

Given concerns over possible bias in the assessment of the three elements of the REF and the cost of preparing the REF return (Farla and Simmonds 2015), we evaluated the implications of the exclusion of different elements from the REF. Since three components of the REF are positively and highly correlated, each of the elements of the REF could be considered redundant and therefore, this article examined the QR funding allocation implications to different panels, UOAs and HEIs when an element (environment, impact and output) of the REF was disregarded from the allocation of the QR funding and the effect on the obtaining the overall REF GPAs. Furthermore, we also use the PCA method to get weights that explain most of the variation in the funding distributed amongst three elements, which suggested that using equal weights to distribute funds explains most of the variation in funding distribution in three pots.

We found that the exclusion of one element from the REF or using equal weights would have benefited (disadvantaged) some HEIs, but at most £46.7 million (out of over £1 billion) would have been reallocated between HEIs when the output element is excluded from the evaluation. Furthermore, when different elements are excluded from the rankings and the weight of the excluded element redistributed between the other two (in proportion to their original weightings) to produce new rankings, these rankings are highly and significantly correlated with the official rankings, suggesting that alternative ways of obtaining composite scores lead to rankings similar to the official one. Overall, the main argument of this article is that given the high cost of preparing REF returns, the potential bias in assessing each component, and the relatively small effect on QR income distribution and universities’ relative rankings of removing some elements of the REF assessment, removal of some elements from the assessment process may be considered for future assessment exercises.

This article does not quantify the bias involved in the evaluation of each element of the REF exercise, and therefore, we do not provide any suggestion about which element should be removed from the REF. Instead, our findings demonstrate that excluding a component from the REF evaluation does not result in significant rank reversals in overall outcomes and reallocation of funds across units and HEIs.

In addition, the assessment of outputs and impact cases in the REF are both based on the submit-to-be-rated methodology from 1986 by which, in essence, the achievements of individuals, not of the organization, are summed up. Based on the definition of organizational evaluation by BetterEvaluation (2021), impact and output evaluations of the REF are based on the achievements of individuals, and if the aim is to evaluate the organizations, then evaluation of the impact and output elements, which are in essence individual achievements, could be removed, and their removal from the evaluation will not result in significant effects as found in this article. Therefore, if the REF aims to evaluate the organizational performance, the choice of the components should be further motivated by and rely on the metrics that evaluate the organization rather than the individual achievements.

Furthermore, if future evaluations include new metrics that aim to measure organizational achievement, these metrics should be carefully chosen to provide a new set of information beyond the existing indicators. Therefore, these indicators should not be highly correlated with the already existing indicator set so that new information is captured through their assessment.

Endnotes

1

There is a significant body of literature on PRFS, and for a review of these systems, the reader is directed to a number of papers and references (Rebora and Turri 2013; Bertocchi et al. 2015; De Boer et al. 2015; Hicks et al. 2015; Dougherty et al. 2016; Geuna and Piolatto 2016; Sivertsen 2017; Zacharewicz et al. 2019, amongst many others).

2

Education is a devolved matter in the UK, and university funding and oversight in 2014 was the responsibility of the Higher Education Funding Council (HEFCE) in England, the Scottish Funding Council (SFC) in Scotland, the Higher Education Funding Council for Wales (HEFCW) in Wales and the Department for Employment and Learning (DELNI) in Northern Ireland. The formulae which converted REF performance into QR funding were different in the different administrations, and this article only examines the QR distribution across English HEIs.

3

An environment that is conducive to producing research of world-leading quality, internally excellent quality, international recognized quality and nationally recognized quality is given 4*, 3*, 2* and 1* scores, respectively. On the other hand, an environment that is not conducive to producing research of at least nationally recognized quality is considered as unclassified.

4

The cost to UK HEIs of submitting to REF, excluding the impact element was estimated at £157 million (Farla and Simmonds 2015, 6). It is further estimated that 12% of time spent at the central level was on the environment template and 17% of time at the UOA level (see Figures 5 and 6 of Farla and Simmonds 2015, respectively). The estimate of £19–27 million is obtained as 12–17% of the overall £157 million non-impact cost of submission. Furthermore, it was found that the panel members spent on average 533 h on panel duties, which represented an estimated cost to the sector of £23 million (see Farla and Simmonds 2015, 40, 41).

5

As stated previously, because the devolved administrations of the UK used different methods to calculate QR income, this article focusses just on English institutions.

6

The scores for each individual output, environment or impact component are not given on the REF 2014 website, www.ref.ac.uk/2014. In other words, the ratings of each research output, research environment element and impact case study are not provided. However, the REF results instead provide the percentage of the overall research elements (i.e., research output, environment and impact) that were rated as 4*, 3*, 2* and 1* and unclassified. Therefore, the weighted average of the research elements (i.e., output, environment and impact) are obtained as follows. If the 35%, 30%, 20% and 15% of the research element of a given submission were rated as 4*, 3*, 2* and 1*, respectively, then the weighted average score of this element would be (35 * 4+30 * 3+20 * 2+15 * 1)/100=2.85.

7

The four main panels are groupings of individual UOAs which broadly speaking encompass medical, and health and biological sciences (Panel A), physical sciences and engineering (Panel B), social sciences (Panel C) and humanities and arts (Panel D).

8

These percentage weights are obtained by (0.65/0.85)×100 and (0.2/0.85)×100, respectively.

9

Note that HEIs within inner and outer London area receive 12% and 8% (respectively) additional QR funding on top of their allocated mainstream QR funding but to examine the effect of the exclusion of alternative scenarios on the allocation of the mainstream QR funding, we do not consider the additional funding allocation that is based on the location of HEI.

Acknowledgements

We would like to thank the editor and three anonymous referees for very constructive and insightful reviews of earlier drafts of this article.

Conflict of interest statement. None declared.

Appendix 1

Table A.1.

Allocation of mainstream QR funding allocation to HEIs with the official and alternative scenarios

Institution	Official (£)	Scenario 1— Official (£)	Scenario 2— Official (£)	Scenario 3— Official (£)	Scenario 3— Official (£)
Anglia Ruskin University Higher Corporation	1,622,895	192,420	59,422	−603,943	−334,196
Arts University Bournemouth, the	118,197	18,469	14,774	−78,616	−40,430
Aston University	4,979,758	32,514	22,038	−129,360	−67,518
Bath Spa University	857,237	90,845	26,662	−281,962	−156,497
Birkbeck College	6,727,502	61,092	171,246	−538,601	−246,422
Birmingham City University	1,704,383	65,665	−186,239	263,985	76,123
Bishop Grosseteste University	57,818	10,224	11,167	−50,323	−25,133
Bournemouth University	2,471,780	55,070	−69,511	24,091	−15,976
Brunel University London	9,206,622	279,285	178,708	−1,087,047	−570,204
Buckinghamshire New University	205,558	23,529	−4,637	−46,755	−29,904
Canterbury Christ Church University	1,216,228	122,552	53,237	−419,683	−227,021
City, University of London	7,905,321	400,555	−109,667	−726,013	−480,793
Courtauld Institute of Art	1,180,656	−51,425	76,890	−49,765	3,887
Coventry University	2,458,307	38,919	−155,277	258,655	86,459
Cranfield University	6,180,196	−235,578	105,717	333,167	244,813
De Montfort University	3,179,060	211,887	−129,321	−221,732	−188,668
Edge Hill University	1,141,009	142,390	−19,099	−303,351	−189,225
Falmouth University	358,169	43,756	−29,422	−39,606	−36,459
Goldsmiths' College	4,377,320	116,852	−96,643	−64,636	−80,727
Guildhall School of Music & Drama	242,154	23,495	−75,799	115,310	35,674
Harper Adams University	246,097	16,548	−3,875	−31,487	−20,467
Imperial College of Science, Technology and Medicine	50,046,827	−1,258,804	−382,687	3,937,198	2,180,722
Institute of Cancer Research: Royal Cancer Hospital	4,103,895	72,379	−71,749	−12,977	−39,056
King's College London	41,332,642	−657,949	8,918	1,582,286	947,415
Kingston University	2,102,493	140,407	24,561	−397,900	−226,548
Leeds Beckett University	1,650,781	109,738	−66,840	−115,149	−97,839
Leeds Trinity University	93,100	15,845	16,230	−75,538	−37,959
Liverpool Hope University	829,581	129,909	93,443	−529,123	−274,730
Liverpool John Moores University	4,395,123	258,287	79,881	−810,947	−448,703
Liverpool School of Tropical Medicine	975,908	−1,208	−44,829	104,985	43,035
London Business School	2,787,787	23,024	218,032	−552,376	−234,214
London Metropolitan University	890,241	103,844	20,456	−299,501	−169,664
London School of Hygiene and Tropical Medicine	9,951,277	−711,574	−582,458	3,059,023	1,569,865
London South Bank University	1,193,809	157,905	−20,399	−338,181	−210,562
Loughborough University	14,223,812	−283,171	−308,809	1,392,657	695,652
Manchester Metropolitan University	4,745,496	309,321	−287,989	−97,888	−184,076
Middlesex University	3,506,781	181,618	−80,441	−259,268	−189,715
Newman University	136,168	22,681	31,723	−127,456	−62,155
Norwich University of the Arts	116,422	16,054	−18,656	3,361	−6,139
Nottingham Trent University	3,508,397	326,685	−68,933	−638,807	−411,011
Oxford Brookes University	3,976,182	442,521	72,944	−1,243,902	−709,903
Queen Mary University of London	19,315,250	404,194	335,395	−1,747,951	−895,910
Roehampton University	2,566,895	217,242	33,881	−606,265	−346,728
Rose Bruford College of Theatre and Performance	60,668	7,180	6,161	−31,511	−16,101
Royal College of Art(The)	1,500,065	−140,240	−102,919	575,855	298,461
Royal College of Music	348,280	38,330	−75,583	78,683	13,927
Royal Holloway and Bedford New College	9,243,679	232,778	96,891	−787,533	−427,316
Royal Northern College of Music	270,785	23,992	−63,940	87,106	24,032
Sheffield Hallam University	3,747,422	159,084	44,339	−488,412	−271,888
Solent University	145,440	23,332	36,360	−139,595	−67,370
St Mary's University, Twickenham	350,290	58,411	60,765	−280,589	−140,792
St. George's Hospital Medical School	1,503,307	220,576	−136,103	−227,457	−195,042
Staffordshire University	626,193	68,556	−61,609	−26,746	−42,841
Teesside University	1,085,407	119,749	−21,115	−243,614	−154,484
The London School of Economics and Political Science	14,428,304	−196,825	188,156	51,121	112,613
The Open University	7,521,123	184,783	175,912	−850,502	−430,372
The Royal Academy of Music	245,544	26,666	−2,264	−59,798	−36,646
The Royal Agricultural University	37,856	6,698	9,464	−37,856	−18,443
The Royal Central School of Speech and Drama	540,676	23,056	−98,433	167,902	57,155
The Royal Veterinary College	3,014,357	−177,384	−91,933	641,323	342,295
The School of Oriental and African Studies	3,537,354	63,265	205,391	−621,618	−281,021
The University of Bath	13,243,620	209,392	−156,407	−153,998	−160,101
The University of Birmingham	26,983,662	521,852	484,144	−2,373,126	−1,203,776
The University of Bolton	397,017	67,751	24,345	−220,440	−120,822
The University of Bradford	2,823,257	159,426	34,054	−465,835	−262,914
The University of Chichester	641,375	75,877	−64,579	−37,819	−50,740
The University of Cumbria	197,724	33,291	20,435	−127,602	−67,170
The University of East Anglia	11,935,131	549,917	72,334	−1,504,101	−865,326
The University of Essex	7,003,419	−39,094	228,121	−424,039	−153,282
The University of Huddersfield	3,325,272	158,867	−44,663	−285,293	−189,616
The University of Hull	5,377,062	385,790	−268,285	−328,997	−313,287
The University of Kent	11,607,649	267,005	−166,279	−271,859	−234,690
The University of Lancaster	15,291,902	−462,411	−29,309	1,193,024	698,610
The University of Leeds	30,883,746	−339,569	−1,013,835	3,134,843	1,426,783
The University of Leicester	13,910,330	744,644	235,587	−2,350,009	−1,298,485
The University of Liverpool	19,621,420	275,615	211,943	−1,153,759	−595,478
The University of Manchester	44,299,217	−1,537,168	−652,384	5,229,119	2,833,377
The University of Reading	13,993,802	−2,504	11,724	−20,588	−7,159
The University of Sheffield	30,115,340	−297	−19,685	45,531	18,558
The University of Surrey	11,006,554	463,899	455,026	−2,165,688	−1,092,790
The University of Warwick	27,203,566	−62,289	966,259	−2,047,730	−799,302
The University of West London	241,250	41,756	42,925	−199,414	−100,174
The University of Westminster	3,396,426	237,591	−7,710	−561,158	−337,986
Trinity Laban Conservatoire of Music and Dance	199,192	14,277	−6,862	−19,155	−14,418
University College London	81,088,324	−3,370,285	−714,947	9,836,506	5,553,467
University for the Creative Arts	334,017	29,843	−49,521	40,032	2,256
University of Bedfordshire	2,069,080	260,034	58,712	−767,017	−431,747
University of Brighton	4,000,024	−49,949	−407,449	1,049,118	447,743
University of Bristol	34,306,380	−376,290	−803,044	2,744,470	1,286,012
University of Cambridge	74,346,811	−2,098,481	1,893,673	800,854	1,304,128
University of Central Lancashire	2,872,634	276,062	90,800	−879,096	−484,574
University of Chester	960,579	145,852	40,433	−447,293	−249,073
University of Derby	712,499	111,966	−5,019	−261,294	−158,001
University of Durham	19,499,795	44,747	−649,139	1,368,612	532,762
University of East London	1,949,327	170,304	−60,080	−278,058	−192,031
University of Exeter	17,977,209	175,679	494,541	−1,553,607	−710,556
University of Gloucestershire	586,397	87,712	−797	−211,828	−126,662
University of Greenwich	2,209,926	317,413	−117,980	−504,582	−352,381
University of Hertfordshire	2,874,591	225,842	193,300	−990,092	−506,020
University of Keele	5,402,925	219,857	22,892	−587,621	−340,408
University of Lincoln	2,578,147	347,490	206,295	−1,315,972	−694,671
University of London Institute in Paris	7,668	1,357	1,917	−7,668	−3,736
University of Newcastle upon Tyne	23,462,169	213,322	−453,434	512,534	107,705
University of Northampton, The	544,299	57,243	21,677	−188,772	−103,104
University of Northumbria at Newcastle	5,369,491	147,571	256,569	−943,459	−450,596
University of Nottingham, The	36,452,153	−1,447,040	−535,555	4,743,660	2,594,892
University of Oxford	82,323,815	−1,952,481	1,360,203	1,659,551	1,583,315
University of Plymouth	6,618,849	551,610	332,297	−2,099,967	−1,107,171
University of Portsmouth	4,633,503	402,772	−202,644	−519,774	−398,396
University of Salford, The	2,992,373	35,009	85,661	−280,257	−129,728
University of Southampton	33,576,701	257,118	−474,892	454,705	63,855
University of Sunderland	1,054,206	95,058	74,681	−401,530	−206,836
University of Sussex	11,042,885	347,485	409,074	−1,777,534	−881,391
University of the Arts, London	2,680,969	78,207	−133,830	114,140	9,646
University of the West of England, Bristol	4,437,156	94,506	−189,833	201,918	37,543
University of Winchester	611,245	78,625	42,104	−287,349	−152,969
University of Wolverhampton	1,604,471	160,149	206,324	−859,725	−422,598
University of Worcester	731,633	106,266	32,766	−333,420	−184,517
University of York	17,506,983	−11,589	−399,935	938,581	385,109
Writtle University College	41,182	7,286	10,295	−41,182	−20,063
York St John University	460,258	76,090	−4,603	−174,857	−106,277

Notes: Official column presents the allocation of mainstream QR funding across HEIs with the official funding allocation.

Scenario 1—Official: This column provides the differences in the mainstream QR funding allocated to the HEIs between Scenario 1 and official case.

Scenario 2—Official: This column provides the differences in the mainstream QR funding allocated to the HEIs between Scenario 2 and official case.

Scenario 3—Official: This column provides the differences in the mainstream QR funding allocated to the HEIs between Scenario 3 and official case.

Scenario 4—Official: This column provides the differences in the mainstream QR funding allocated to the HEIs between Scenario 4 and official case.

A positive (negative) figure in changes columns suggests that the HEI received relatively more (less) QR funding with the respective alternative scenario compared to the official case.

Table A.2.

GPA scores and respective rankings of HEIs with the official case and Scenarios 1, 2, 3 and 4

Institution name	GPA— REF2014	GPA— Scenario 1	GPA— Scenario 2	GPA— Scenario 3	GPA— Scenario 4	Rank— REF2014	Rank— Scenario 1	Rank— Scenario 2	Rank— Scenario 3	Rank— Scenario 4
Anglia Ruskin University	2.367	2.407	2.368	2.267	2.308	92	92	90	91	91
Arts University Bournemouth	2.310	2.400	2.337	2.028	2.154	95	93	92	102	99
Aston University	3.053	3.011	2.993	3.293	3.170	35	35	33	33	34
Bath Spa University	2.523	2.569	2.470	2.532	2.505	84	84	86	87	87
Birkbeck College	2.966	2.931	2.933	3.125	3.046	43	45	39	46	44
Birmingham City University	2.642	2.643	2.528	2.899	2.746	77	76	81	62	67
Bournemouth University	2.719	2.702	2.657	2.900	2.800	65	71	67	61	63
Brunel University London	2.674	2.638	2.654	2.807	2.744	73	77	68	70	68
Canterbury Christ Church University	2.382	2.411	2.351	2.384	2.370	90	91	91	89	90
City University London	2.947	2.940	2.883	3.110	3.016	46	44	43	47	47
Courtauld Institute of Art	3.480	3.389	3.500	3.658	3.595	1	1	1	4	1
Cranfield University	2.906	2.847	2.838	3.206	3.055	50	53	49	43	43
De Montfort University	2.674	2.676	2.594	2.851	2.744	74	74	76	67	69
Edge Hill University	2.224	2.265	2.215	2.146	2.174	97	98	97	98	97
Falmouth University	2.045	2.019	1.992	2.228	2.131	105	108	108	94	100
Goldsmiths' College	2.899	2.869	2.848	3.090	2.990	51	52	47	49	50
Guildhall School of Music & Drama	2.435	2.459	2.144	3.041	2.669	89	90	98	53	77
Harper Adams University	2.663	2.648	2.637	2.760	2.709	76	75	71	74	76
Heythrop College	2.824	2.740	2.680	3.357	3.079	55	66	63	28	42
Imperial College London	3.361	3.280	3.280	3.740	3.552	3	4	7	3	3
Institute of Cancer Research	3.407	3.361	3.292	3.781	3.579	2	2	5	2	2
Institute of Zoology	3.121	3.125	2.994	3.404	3.234	22	12	32	26	28
Keele University	2.880	2.874	2.841	2.983	2.924	52	50	48	57	54
King's College London	3.230	3.161	3.158	3.564	3.398	8	9	9	11	9
Kingston University	2.705	2.714	2.661	2.784	2.733	69	69	65	71	70
Lancaster University	3.150	3.066	3.086	3.498	3.329	19	25	17	17	14
Leeds Beckett University	2.160	2.199	2.099	2.204	2.160	100	100	100	95	98
Liverpool Hope University	2.206	2.307	2.260	1.834	2.008	98	97	95	105	103
Liverpool John Moores University	2.793	2.804	2.766	2.829	2.803	57	54	55	68	62
Liverpool School of Tropical Medicine	3.096	3.037	2.978	3.505	3.289	26	30	35	16	21
London Business School	3.287	3.227	3.342	3.305	3.322	7	7	2	32	15
London School of Economics and Political Science	3.350	3.281	3.312	3.607	3.487	4	3	3	9	7
London School of Hygiene & Tropical Medicine	3.192	3.064	3.054	3.813	3.502	11	26	21	1	4
London South Bank University	2.519	2.571	2.442	2.569	2.515	86	83	87	84	86
Loughborough University	2.952	2.894	2.896	3.221	3.088	44	48	40	42	40
Manchester Metropolitan University	2.744	2.751	2.659	2.918	2.810	62	63	66	58	58
Middlesex University	2.584	2.555	2.547	2.740	2.661	80	85	78	76	78
Newcastle University	3.086	3.048	3.016	3.339	3.206	29	29	30	31	30
Norwich University of the Arts	2.685	2.752	2.531	2.870	2.728	72	62	80	64	73
Nottingham Trent University	2.588	2.607	2.541	2.648	2.603	79	80	79	81	81
Open University	2.908	2.886	2.872	3.046	2.974	49	49	45	51	52
Oxford Brookes University	2.665	2.716	2.652	2.567	2.601	75	68	69	85	82
Queen Mary University of London	3.181	3.147	3.141	3.351	3.265	12	10	10	30	24
Roehampton University	2.826	2.872	2.777	2.825	2.804	54	51	54	69	61
Rose Bruford College	2.288	2.391	2.235	2.156	2.186	96	94	96	97	96
Royal Academy of Music	2.753	2.780	2.716	2.771	2.748	60	58	59	72	66
Royal Agricultural University	1.394	1.376	1.543	1.101	1.284	110	111	110	110	110
Royal Central School of Speech and Drama	3.030	3.017	2.821	3.537	3.241	37	34	51	12	27
Royal College of Art	3.078	2.951	2.962	3.654	3.371	31	42	38	5	10
Royal College of Music	3.014	3.051	2.792	3.427	3.163	39	28	53	23	37
Royal Holloway, University of London	3.088	3.056	3.047	3.258	3.172	27	27	22	37	33
Royal Northern College of Music	3.082	3.114	2.853	3.527	3.247	30	16	46	14	26
Royal Veterinary College	3.119	3.007	3.035	3.582	3.358	24	36	25	10	11
School of Oriental and African Studies	2.812	2.787	2.808	2.881	2.852	56	57	52	63	56
Sheffield Hallam University	2.763	2.757	2.725	2.864	2.807	59	61	58	65	59
Southampton Solent University	1.628	1.645	1.794	1.206	1.449	109	109	109	109	109
St Mary's University, Twickenham	1.981	2.089	2.050	1.564	1.763	108	107	104	108	108
St. George's, University of London	2.993	3.100	2.832	3.096	2.984	41	20	50	48	51
Staffordshire University	2.193	2.218	2.123	2.293	2.222	99	99	99	90	95
Teesside University	2.575	2.636	2.502	2.592	2.554	81	78	82	83	83
Trinity Laban Conservatoire of Music and Dance	2.784	2.799	2.681	2.985	2.859	58	55	62	56	55
University College London	3.219	3.117	3.138	3.649	3.440	10	14	11	6	8
University for the Creative Arts	2.715	2.718	2.569	3.042	2.846	66	67	77	52	57
University of Bath	3.174	3.132	3.106	3.431	3.298	14	11	13	22	20
University of Bedfordshire	2.523	2.575	2.502	2.444	2.466	85	82	83	88	88
University of Birmingham	3.071	3.028	3.033	3.263	3.169	33	32	27	36	35
University of Bolton	2.038	2.134	2.051	1.775	1.887	106	105	103	106	106
University of Bradford	2.944	2.945	2.881	3.085	3.001	47	43	44	50	49
University of Brighton	2.839	2.796	2.709	3.240	3.021	53	56	60	40	46
University of Bristol	3.176	3.114	3.096	3.510	3.340	13	17	14	15	13
University of Cambridge	3.327	3.240	3.286	3.629	3.489	6	6	6	8	6
University of Central Lancashire	2.512	2.529	2.482	2.542	2.516	87	86	85	86	85
University of Chester	2.075	2.153	2.083	1.865	1.953	103	102	101	104	105
University of Chichester	2.498	2.528	2.420	2.601	2.526	88	87	88	82	84
University of Derby	2.071	2.146	2.041	1.956	1.989	104	103	105	103	104
University of Durham	3.140	3.088	3.069	3.426	3.279	20	22	19	24	22
University of East Anglia	3.112	3.099	3.062	3.254	3.175	25	21	20	38	32
University of East London	2.711	2.743	2.651	2.766	2.718	67	64	70	73	75
University of Essex	3.049	2.989	3.030	3.238	3.154	36	38	28	41	38
University of Exeter	3.078	3.029	3.047	3.267	3.177	32	31	23	35	31
University of Gloucestershire	2.373	2.483	2.307	2.257	2.275	91	89	94	93	92
University of Huddersfield	2.632	2.629	2.621	2.664	2.646	78	79	73	80	79
University of Hull	2.699	2.704	2.626	2.851	2.758	70	70	72	66	65
University of Kent	2.952	2.925	2.896	3.144	3.042	45	46	41	45	45
University of Leeds	3.130	3.067	3.046	3.475	3.299	21	24	24	18	19
University of Leicester	2.925	2.915	2.892	3.026	2.971	48	47	42	54	53
University of Lincoln	2.538	2.593	2.598	2.266	2.402	83	81	75	92	89
University of Liverpool	3.060	3.019	3.004	3.283	3.169	34	33	31	34	36
University of Manchester	3.157	3.072	3.084	3.530	3.347	17	23	18	13	12
University of Northampton	2.091	2.106	2.068	2.106	2.090	101	106	102	100	101
University of Northumbria at Newcastle	2.710	2.683	2.729	2.731	2.731	68	73	57	77	71
University of Nottingham	3.086	3.007	3.018	3.435	3.264	28	37	29	21	25
University of Oxford	3.335	3.251	3.292	3.639	3.498	5	5	4	7	5
University of Plymouth	2.735	2.741	2.737	2.714	2.724	64	65	56	78	74
University of Portsmouth	2.752	2.774	2.662	2.907	2.805	61	59	64	59	60
University of Reading	3.028	2.987	2.979	3.243	3.135	38	39	34	39	39
University of Salford	2.539	2.503	2.500	2.714	2.627	82	88	84	79	80
University of Sheffield	3.172	3.121	3.107	3.444	3.306	15	13	12	20	18
University of Southampton	3.151	3.106	3.088	3.405	3.275	18	19	16	25	23
University of Surrey	2.977	2.958	2.975	3.025	3.005	42	41	37	55	48
University of Sussex	3.005	2.972	2.977	3.151	3.080	40	40	36	44	41
University of the Arts, London	3.121	3.107	3.034	3.352	3.221	23	18	26	29	29
University of the West of England, Bristol	2.698	2.691	2.615	2.906	2.785	71	72	74	60	64
University of Warwick	3.220	3.171	3.200	3.384	3.309	9	8	8	27	16
University of Westminster	2.739	2.772	2.697	2.753	2.729	63	60	61	75	72
University of Winchester	2.318	2.380	2.315	2.175	2.231	93	95	93	96	93
University of Wolverhampton	2.313	2.328	2.378	2.127	2.231	94	96	89	99	94
University of Worcester	2.078	2.138	2.031	2.039	2.034	102	104	107	101	102
University of York	3.167	3.117	3.094	3.456	3.307	16	15	15	19	17
Writtle College	1.281	1.424	1.501	0.431	0.870	111	110	111	111	111
York St John University	2.034	2.173	2.037	1.691	1.831	107	101	106	107	107