The effects of funding and co-authorship on research performance in a small scientific community

Abstract

The evaluation of research performance increasingly relies on quantitative indicators determined by national science policies. We focus on two dimensions of research performance—productivity and excellence—as defined in the evaluation methodology of the Slovenian Research Agency. Our analysis focuses on the effects of two science policy factors—co-authorship collaboration and researcher funding—on the productivity and excellence of Slovenian researchers at the level of research disciplines. A multilevel analysis using a hierarchical linear model with regression analysis was applied to the data with several nested levels. As many variables have a semi-continuous distribution, a statistical model was used to address them. The results show a very strong positive effect of international co-authorship collaboration on productivity and excellence, while fragmentation of funding shows a negative impact only on excellence. We also include interviews with excellent Slovenian researchers regarding their views on scientific excellence and quantitative indicators.

1. Introduction

In modern societies, the results of R&D are closely related to national development, economic growth and competitiveness. In the last decade, we have witnessed both a national and a transnational shift to increasing use of quantifiable indicators of scientific productivity (SP) and excellence, which are supposed to both measure and promote SP and the excellence of such productivity. While some attempts have been made to theoretically conceptualise and explore these two dimensions of scientific performance, it is difficult to capture them in substantive definitions since the two categories are context-dependent and very changeable over time (Barré 2010; Hellström 2010; Ochsner et al. 2013; Tijssen et al. 2002). Let us consider the example of scientific excellence: one can assume that research findings are more likely to gain credibility if they are driven by excellent research. If everybody agrees that excellence in research is important, crucial questions then arise—how we separate the »good« from the »bad« in science; or which criteria should be used to evaluate scientific research. This produces more questions than answers. Sometimes difficulties arise when trying to find a consensus on the meaning of scientific excellence (SE) and if and how it differs from SP. Some scholars conceive of research impacts as part of research quality (Yates 2005) while others note that quality and impact are two different elements that constitute SE (Grant et al. 2010). This is further complicated by what the scientific community considers as excellent and desirable and how this differs from what policymakers see as desirable and excellent in the context of national competitiveness.

In this regard, the notion of SE has undergone a radical shift at the transnational level. At the EU level, the understanding of excellence has shifted from a relatively fuzzy concept embedded in the research community and revealed through peer reviews to a more sharply defined one, connected with breakthrough research (Sørensenet al. 2016) which policymakers attempt to measure and promote with quantifiable indicators. Indeed, we can observe variations of this trend all over the world. Similarly, in Central and Eastern European (CEE) countries, various research agencies, ministries of science and other governmental institutions have established R&D evaluation mechanisms to monitor and influence the quality of scientific output, also in order to provide more objective tools for use in R&D decision-making processes (through the allocation of research funds, support for R&D human resources, grants for young researchers, etc.). Following the political changes in the 1990s, national R&D policies in this part of Europe have made great efforts to increase the accountability and transparency in science, policy-making and funding bodies, and to promote production and excellence in their national science systems. Quantifiable indicators such as publication and citation counts are now extensively used to evaluate both dimensions of scientific performance (Frankel and Cave 1997; Kovács and Kutsar 2012; Mali 2011; Mayntz et al. 1998). Ultimately, if the aim of measuring those quantitative indicators is also to steer and promote their outputs, then it makes sense to know which factors influence them and in what ways. Two factors of national science policies are particularly relevant in this regard, namely: (1) funding and (2) scientific collaboration.

1. The allocation of funds is increasingly linked to R&D performance indicators. Especially in small countries where R&D funding resources are relatively scarce and where there are only a few funding agencies, an efficient R&D funding policy that is able to mitigate the effects of smallness is required. Although experts’ opinions do not always agree about the direct impact of such a ‘size effect’ (Horta and Lacy 2011) on performance indicators, our assumption is that a fragmentation of R&D money is negative for science. Namely, as many analysts have warned, the inefficient allocation of financial resources can hinder the quality of science (Hung and Shiu 2014).

2. R&D performance indicators depend more and more on networking in science. The various forms of collaboration among researchers are one of the most visible characteristics of modern science. John Ziman states: ‘Modern science is becoming a de facto collectivized enterprise’ (Ziman 1994: 218). Most bibliometric analyses have confirmed that co-authorship collaboration in science has a positive effect on R&D performance indicators (Andrade et al. 2009; Guimera et al. 2005; Ponomariov and Boardman 2010). In our previous bibliometric analyses where we dealt with the dynamics of co-authorship within small scientific communities, we came to the same results (Mali et al. 2010, 2012; Kronegger et al. 2012; Ferligoj et al. 2015).

In this article, we focus on the case of Slovenia as a country with a small scientific community in the CEE region, and evaluate the current state of the research performance of its scientific system after changes were made in the Slovenian R&D policy. In particular, we concentrate on analysis and explanation of research productivity and research excellence as defined and used in the evaluation methodology of the Slovenian Research Agency (SRA) through a number of quantitative bibliometric indicators. In this way, we evaluate the research productivity and excellence of Slovenian researchers according to these bibliometric indicators, and also explore the impact of two primary scientific policy factors, namely the state of research funding and research collaboration through co-authored publications, on scientific performance. In our view, in the small Slovenian scientific community with a relatively short audit culture tradition, R&D evaluation based on bibliometric indicators can play an important role in improving SP and excellence. This short tradition refers to the use of strict institutional R&D evaluation procedures which in old Western European democracies were broadly established a long time ago. R&D evaluation procedures in Slovenia are limited to quite a small number of tasks (ex ante evaluation of R&D project proposals, evaluation of individual research track records in habilitation processes, etc.). For that reason, the recent introduction of quantitative performance indicators could help increase the transparency of R&D policy decision-making, ensuring that public money is properly spent and that the administrative burden of the R&D management process is minimised.

We are, however, aware that the use of quantitative (bibliometric) measures of scientific quality is not independent of specific values, norms and interests, and in the context of national R&D policies can easily reflect a distorted reality. Therefore, we expand our bibliometric analysis with the qualitative results of interviews with a small number of excellent Slovenian scientists who provide a wider view on research excellence, bibliometric indicators and the policy factors influencing them.

Our contribution is divided into four main sections. In the first part, we present the situation in Slovenia concerning the use of R&D evaluation with quantitative indicators and its impact on publication productivity and excellence. The second section provides the conceptual and methodological basis for our bibliometric analysis of the data on Slovenian scientists and their publication productivity and excellence, as defined by the SRA. In the third section, the results of the bibliometric analysis are presented and discussed. In the fourth section, we provide the results of the interviews with some excellent scientists about their perceptions of excellence, quantitative indicators and co-authorship collaboration, followed by some short concluding remarks.

2. The role of R&D evaluation instruments in Slovenian science policy

Throughout the history of modern science, it has been expected that academic scholars with the best scientific results would gain better access to (symbolic and material) resources and rewards that enhance their career track and reputation in the scientific community. This expectation has been consistent with the normative ethos of science, which asserts that the assessments of research quality and the distribution of rewards in the scientific community are governed by the meritocratic application of universal criteria (Merton 1973; Ziman 1994; Whitley 2007). The reorganisation of science and society in the second half of the twentieth century forced different stakeholders (funding agencies, governments, managers of research institutions, deans, etc.) to face the evaluative challenge of how to determine, recognise and compare SE, scientific productivity, scientific impact, etc. This R&D policy agenda changed radically in the mid-1970s with the introduction of quantitative R&D evaluation instruments.

It seems that at the turn of the twentyfirst century the trend to enhance quantitative R&D evaluation became even stronger. R&D evaluation has since been extended to a broad range of activities; it takes place at different organisational levels, follows different goals and attracts the attention of different stakeholders (Ernø-Kjølhede and Hansson 2011). ‘In Europe, we witness the triumph of science and technology (S&T) indicators—not only of bibliometric indicators—in the context of the encompassing need for assessments and the striving for evidence-based policies’ (Barré 2010: 229).

In Slovenia, an orientation to R&D evaluation based on quantitative indicators appeared in the early 1990s after Slovenia had become an independent state. In the old political regime, Slovenia as part of former Yugoslavia had to endure a long period of voluntary political regulation of science. Under the former regime, the publish or perish rationale based on an objective type of R&D evaluation was never fully realised (Mali 2011, 2013). The situation changed in the 1990s with the reorganisation of R&D policies which brought in R&D evaluation procedures that were previously entirely absent. More recently, after twenty-five years of slow transformations, the R&D evaluation system in Slovenia has been playing a central role in shaping the conditions of scientific life; especially in the public R&D sector (The public research sector contains most of the R&D capacity in Slovenia. It consists of universities and public research institutes. Four public universities cover a wide spectrum of scientific disciplines, organised in small research groups that are suited to educational needs. In contrast, public research institutes are generally more specialised and conduct research in a limited number of disciplines within research groups of a similar size as in the universities.).

In Slovenia, the task of R&D evaluation is in the hands of the SRA, which is also the only institution for public research funding in Slovenia. The Agency was established in the early 2000s (The SRA serves as an intermediary organisation between the public research sector and politics. Decision-making in research agencies and research councils assures that external (policy) imperatives are integrated into actual research practice, namely that external social demands and expectations are mediated to the producers of scientific knowledge (Braun 1997).).

Because of its exclusive role in funding public research in Slovenia, it is also directly responsible for evaluating all types of research project proposals: basic and applied science projects, post-doctoral projects, and science programmes. For scientists working in the public R&D sector, such projects and programmes provide a very important institutional framework for day-to-day research work. This situation is in line with Ziman's thesis that: ‘…projects and programs emerge as the nodal points of modern scientific culture, where its personal, material, social and epistemic dimensions intersect’ (Ziman 2002: 186).

The SRA provides grants on the basis of proposed programmes and projects. Expert bodies at the SRA evaluate the proposed programmes and projects in relation to the overall budget for the public sector in Slovenia during the monitored period. As the only provider of funding for public R&D activity in Slovenia, the SRA is currently funding 335 long-term programmes and 470 short-term projects (SICRIS 2016). The main problem is that in some scientific fields SRA funds a large number of long-term research programmes, but does so with very low budgets, even though these programmes would need greater funding in order to successfully produce excellent R&D results. In this regard, we can talk about a ‘fragmentation’ of funding in the sense that individual researchers are forced to piece together their funding from diverse SRA projects and programmes, and to combine these with teaching hours as well as with funds from international and EU projects, all simply to be able to receive full income. This situation also causes a fragmentation of the researcher's efforts since they are forced to divide their time among different research themes and priorities, while simultaneously needing to keep an eye on acquiring new sources of funding once the current ones come to an end. Further, the formal eligibility criteria to become a principal investigator (or even a member) of a research programme in these scientific fields (disciplines) are very low (the criteria for short-term projects are stricter) (Regarding evaluation, the funding for projects should generally be subjected to more extensive assessments combining both qualitative and quantitative mechanisms. The funding for programmes, which usually constitutes a smaller percentage of researcher salaries, can be performed with quantitative assessments.).

If we compare this situation with other European countries, we should mention some interviews with high-performing groups of Danish and Swedish researchers which indicate that having secure and more concentrated funding, not having to constantly worry about acquiring new funding and being able instead to focus on doing research are all major contributors to research excellence (Young et al. 2015).

Another factor that has recently emerged as an important element influencing R&D performance is scientific collaboration, usually operationalised as the co-authorship of articles. On one hand, modern scientific challenges demand the combination of diverse expertise and research infrastructures. On the other, international and inter-institutional cooperation is also mandated by both the modern scientific collaboration structures and practices as well as the funding agencies that demand the cooperation of different sectors and stakeholders. Co-authorship, especially in the form of both international and inter-institutional collaboration, is thus a growing trend (Sonnenwald 2007). For example, 64 per cent to 84 per cent of papers from twenty-five universities in the European Research Area that published the most scientific articles between 2007 and 2011 were produced by inter-institutional co-authorship (EC 2013). The influence of this global trend is partially also reflected in Slovenian science policy. The Resolution on the research and innovation strategy of Slovenia 2011–2020 thus mentions that one indicator of the goal of achieving more excellent, internationally recognisable research should be the number of international scientific publications (scientific publications in co-authorship with researchers from abroad) per million inhabitants.

As in other European countries, the administration of the SRA is increasingly inclined to use bibliometrics to assess research performance. The research agencies and councils of many European countries face requirements to collect more performance information and tie it to decision-making. The latter is connected to the growing pressure on research agencies and councils to demonstrate programme success in terms of the broad impacts of the results of funded research. SRA is no exception in efforts to place greater importance on bibliometric indicators in R&D evaluation processes.

Another positive aspect of bibliometrics is the transparency of decision-making. Due to the small size of the Slovenian scientific community, bibliometrics enables evaluators to better avoid conflicts of interest. Slovenia is not only a very small country but is to use Thorsteinsdottir’s term, a ‘mini-country’ (Thorsteinsdóttir 2000: 434). A country’s small size does not necessarily lead to a higher level of transparency if R&D evaluations are based only on qualitative peer review. For that reason, R&D policy actors have in the last few years pointed to the use of bibliometrics as one of the most important instruments for increasing the transparency and accountability of R&D policy decision-making in Slovenia (Demšar 2013).

However, the use of quantitative indicators to measure R&D performance is also the subject of strong criticism in Slovenia. Unfortunately, the legitimate criticism of the possible misuses of quantitative evaluation of R&D performance around the world (e.g. Hicks et al. 2015; Wouters et al. 2013) is also abused by some lobbying groups in the small scientific community in Slovenia. In our view, the Slovenian scientific community nevertheless needs to rely on a firm basis of bibliometric indicators. Of course, these must be used cum grano salis, that is, they must be combined with qualitative (peer review) evaluations.

By (conceptually) distinguishing publication productivity from publication excellence, we empirically investigate whether the two selected policy factors, that is, funding and co-authorship collaboration, have an impact on productivity alone, or whether they are also necessary ingredients of publication excellence. Our approach is flexible enough to allow more general conclusions regarding the ‘impact’ of the two factors on SP and SE in Slovenia. Drawing on our theoretical assumptions, we formulate two basic hypotheses:

H1: The fragmentation of funding has a negative effect on the SP and SE of researchers.

H2: The collaboration of researchers within and beyond national borders has a positive effect on their SP and SE.

3. Empirical investigation of the factors affecting SP and excellence

3.1 Data

The data used in our empirical analysis are based on complete scientific bibliographies published in the period between 2000 and 2010 for all 12,164 Slovenian researchers with a research ID at the SRA who published in the observed time period. The data on bibliographies were gathered from the Co-operative On-Line Bibliographic System & Services (COBISS). Personal information on researchers was obtained from the Current Research Information System (SICRIS), while data on the distribution of funds among scientific disciplines were provided by the SRA. COBISS and SICRIS are interconnected information systems and are both maintained by the Institute of Information Science in Maribor (http://www.izum.si). The SICRIS system provides information about the education, positions and employment of researchers registered with the agency, information on research groups and institutions as well as information on both projects and programmes financed with public money. It is therefore an integral system which significantly supplements COBISS by providing more information. COBISS allows access to the complete bibliography of every researcher in Slovenia. Before starting the bibliometric analysis, we excluded researchers who belong to research disciplines with a small number of active researchers. Therefore, the analysis was performed on 12,105 researchers from sixty-eight research disciplines.

The scientific disciplines are nested within the main research fields. We followed the SRA classification system, which divides the R&D landscape in Slovenia into six main scientific fields: natural and mathematical sciences, technical sciences, medical sciences, biotechnical sciences, social sciences, and the humanities.

3.2 Operationalisation of the variables

Our analysis focuses on two dependent variables measuring research performance: publication productivity and publication excellence, as they are defined and implemented by the quantitative indicators chosen by the SRA in their research performance evaluation methodology, and two policy factors that could stimulate or inhibit their results, described in the theoretical part above: (1) the increase of collaboration in modern R&D activity; and (2) the fragmentation of funding for R&D activity. In the model, several control variables are also used.

The quantitative indicators of research performance that we summarise below are taken from the ‘Rules on the Procedures of the (Co)financing and Monitoring of Research Activities Implementation’, an official SRA document for evaluating research performance in Slovenia (see https://www.arrs.gov.si/en/akti/prav-sof-ocen-sprem-razisk-dej-sept-11.asp).

Dependent variables (For an exact definition of the two scores, see Articles 37 and 39 at https://www.arrs.gov.si/en/akti/prav-sof-ocen-sprem-razisk-dej-260111.asp.):

• SP is represented by the sum of scores assigned to all publications produced by an individual researcher. The scores are determined according to publicly available rules defined by expert bodies at the SRA and available in the scoring and evaluation methodology linked above. The publications consist of an extensive group of articles in scientific journals, books (monographs), book chapters, conference papers in conference proceedings and other types of publications produced by the researchers. The scoring of SP also takes into account the differences in publication practices specific to each scientific field. The differences are reflected in the types of publication that have the biggest impact in a field, that is, publications in top quality international scientific journals in the natural sciences, technical sciences, biotechnology and medicine, scientific monographs and scientific journals in the social sciences, and also other types of scientific and expert publications in the humanities. The type of the individual scientific publication, which is often closely linked to expert work, is also taken into account. Given the rules described in the SRA methodology, the total score of SP for all fields is calculated by taking into account the contribution of the following items:

  1. scientific paper in a journal indexed by SCI Expanded and SSCI;

  2. scientific paper in the category A or B journal in ERIH or in a A&HCI journal;

  3. scientific monograph of more than fifty pages;

  4. patent, variety or breed;

  5. being editor of a journal or a monograph or member of an editorial board;

  6. extensive scientific paper or a chapter (more than fifty pages) in a scientific monograph; and

  7. scientific paper or chapter in a scientific monograph.

  The total score for each researcher is calculated in the following way according to SRA criteria, taking into account some field-specific differences:

  • For the natural sciences, the technical sciences, medicine and biotechnology the score consists of the total score from the items listed above and the remaining score of SE (see below). The latter should not exceed 15 per cent of the total score of a researcher.

  • For the social science and humanities: the score consists of the total score plus a score from publications in scientific journals in category C of ERIH or journals indexed by other selected international bibliographical databases if the journal is published in English, French, German, Russian, or Spanish, the score from other scientific monographs above fifty pages published by an international or domestic publisher, the score from other scientific monographs of 20–50 pages if published by foreign publishers, and the remaining score of SE. The latter should not exceed 15 per cent of the total score.

  • Solely for the humanities the score of expert excellence (expert articles, popular articles, etc.) will be added to the total score. The expert score should not exceed 15 per cent of the total score.

  Since the actual value of SP is field-specific, the field affiliation of each individual researcher is used in the model as a control variable.

• SE is determined by the SRA in the same way as the SP score but only summed over selected high-quality publications:

  • papers in the upper quarter of SCI journals;

  • papers in the three upper quarters of SSCI and A&HCI journals as well as journals in the categories A and B of ERIH;

  • patents, varieties, or breeds; and

  • scientific monograph published by an international science publishing house on the SRA list, and solely for humanities a scientific monograph published by a foreign or domestic publishing house.

Dependent variables (publication productivity and excellence) are typical semi-continuous variables (see Fig. 1). Such variables exhibit a mixture of: (1) zeros, corresponding to researchers with no (excellent) publications; and (2) a continuous distribution of positive values, corresponding to the productivity or excellence of researchers. The semi-continuous response can be viewed as the result of two processes, the first determining whether the researcher has begun publishing or not, and the second determining the researcher’s number and quality of publications. Because the dependent variables are severely right-skewed, a logarithmic transformation was applied for values above zero.

Figure 1

Semi-continuous distribution of SP and SE with an added constant and treated by a log transformation.

Explanatory variable measuring the allocation of funding for R&D

• Fragmentation of funds: is measured at the level of research disciplines. It is defined as the number of researchers employed in a research discipline divided by the amount of Full Time Equivalent (FTE) of funds for the discipline (FTE denotes how many full-time researchers can be financed by a certain amount of funds.). The FTE is an instrument the SRA uses to allocate financial resources to scientific disciplines. Variable value 1 represents the situation where all researchers employed in the discipline are fully funded only by the SRA, while larger numbers mean a higher level of fragmentation.

Explanatory variables measuring the co-authorship activities of researchers

• Collaboration outside the SRA is an index of collaboration computed as the sum of all contributions made by co-authors who were not members of the SRA to each publication of a researcher. The sum of the contributions of all authors to one publication equals 1 (Kronegger et al. 2015).

• Betweenness is one of the centrality measures in a network. It measures the importance of each researcher in the co-authorship network. It counts how many times a researcher is on the shortest path between any pair of researchers from the network. To obtain a normal distribution and retain the part of the variable with zero values, a constant of size 10⁻⁹ was added to all values. This variable was transformed using a logarithmic transformation.

• Zero betweenness: researchers with a betweenness value equal to 0 are researchers who do not have any or have only one co-author (this means they are not on any of the shortest paths between any two researchers in the network). Authors with zero betweenness possibly collaborate only with authors that do not have a SRA research ID. As with both dependent variables, the betweenness variable has a semi-continuous distribution. There are 39 per cent of researchers with zero betweenness. A dummy variable was defined that splits zero values (1) from other non-zero values (0) of the betweenness variable.

Controlling variables

• The Research Field of each researcher was obtained from the SICRIS database. The research field and the research discipline are both chosen by researchers at the time of joining the SRA. Changes of disciplines are rare and have no significant impact on the results of the study. Research field affiliation is used to control differences in the definition of dependent variables between the social sciences, humanities and the four other research fields.

• Gender of researcher is defined as a dichotomous variable: female (one) and male (two).

• Researcher’s scientific age is defined as the year of a researcher’s first publication available in the COBISS database, divided by 100. The variable is a proxy for a researcher’s scientific age.

• PhD is a dichotomous variable indicating whether a researcher has a doctoral degree. If a researcher holds a PhD, the value of the variable is 1, otherwise it is 0.

3.3 Methods

Multilevel analysis employing a hierarchical linear model with regression analysis was used for data with a number of nested levels. As several variables have a semi-continuous distribution, appropriate statistical models are used which are common in actuarial studies where they are called frequency-severity models (Bowers 1997; Frees 2009), referring to the components of frequency (whether an insurance claim has occurred) and severity (the claim amount). In econometrics, corresponding models are known as two-part models (Jones 2000). In order to apply the two-part model, a binary variable is defined indicating whether or not the i-th subject has a non-zero value on the dependent variable. The two-part model then consists of: (1) A binary regression model with the set of explanatory variables. (2) Conditional on the first (i.e. for the subset of subjects with non-zero values on the dependent variable), a regression model of the set of explanatory variables. The two sets of variables do not need to be identical, although there is usually a considerable overlap of explanatory variables. The assumption is typically made that the regression coefficients in the frequency model are not related to the regression coefficients in the severity model. The joint likelihood of the data can thus be separated, meaning that the two model components can be estimated separately (Frees 2009).

Since we want to control the effect of belonging to a particular research field on a researcher's SP and excellence, the research fields are represented through fixed effects (indicator variables) in the models. On the other hand, because of the large number of disciplines (sixty-eight) we are not interested in estimating and comparing the effect of each discipline. The disciplines are therefore represented in the models through random intercepts. The model assumes that the random effects for disciplines are normally distributed and that the random effects in the frequency model are uncorrelated to the random effects in the severity model (for a model that allows correlation, see Olsen and Schafer 2001). All models were tested for multicollinearity.

3.4 Findings and discussion

Descriptive statistics for each variable are reported in Table 1. Due to the semi-continuous distribution of the two dependent variables which delineate the two basic models—one for SP and one for SE—the results of each basic model are presented through two sub-models. The first represents the estimated frequency parameters for the dichotomous part of the variable (publishing or not publishing at least one scientific publication or at least one excellent publication), while the second represents severity parameters for the continuous part of the dependent variable (reaching a higher level of productivity among those who publish, or reaching a higher level of excellence among those who have already published an excellent contribution). In the Tables 2 and 3 the pseudo R² (likelihood ratio based) is given for all four models. It is relatively high (between 35% and 42%). Explained variance components on discipline level are higher than the residuals.

Table 1

Descriptive statistics.

Variables		Research field
		Natural	Technical	Medical	Biotechnical	Social	Humanities
Excellence.bin	per cent of bin >0	84.2	66.1	83.6	74.7	81.6	97.4
Excelence	$\stackrel{-}{x}$	366.836	247.402	195.315	185.906	241.078	474.74
(for values >0)	SD	573.976	398.844	283.753	253.589	274.257	744.339
Product.bin	per cent of bin >0	61.9	37.9	50.9	46.8	27	62.6
Product	$\stackrel{-}{x}$	269.732	174.523	111.88	109.39	102.956	306.31
(for values >0)	SD	454.996	283.223	169.69	137.005	118.058	503.275
Fragment. of	$\stackrel{-}{x}$	6.286	14.099	11.886	7.715	10.528	4.939
Funds	SD	3.946	8.852	4.383	6.708	5.483	1.573
Collaboration	$\stackrel{-}{x}$	0.059	0.043	0.063	0.046	0.036	0.017
Outside ARRS	SD	0.068	0.062	0.069	0.055	0.058	0.042
Betweenness*10−4	$\stackrel{-}{x}$	4.00	4.43	3.64	3.36	4.34	3.27
(for values >0)	SD	9.05	10.7	8.06	8.55	8.34	4.93
Betwenness.0	per cent of betw = 0	31.4	35.7	25.7	24.6	45.5	75.9
Gender	per cent of male	57.6	81.6	47.8	51	52.5	47.8
Age	$\stackrel{-}{x}$	19.906	19.925	19.907	19.898	19.876	19.832
	SD	0.136	0.119	0.105	0.13	0.128	0.177
PhD	per cent of yes	75	58.5	52.9	68.8	66.1	71.2

Table 2

Estimated parameters for frequency and severity in the SP model. The left side presents the results of the logistic regression on the complete sample (frequency). The right side presents regression results of the sub-sample (variable SP > 0) (severity). The reference category for the research field is ‘Natural sciences and mathematics’. Bold values are significant at 0.05 level.

		Frequency				Severity
		est.	s.e.	z value	odds r.	est.	s.e.	t value
(Intercept)		−6.07	4.91	−1.23	0.00	25.83	1.61	16.06
Fragmentation of funds		−0.02	0.01	−1.54	0.98	−0.01	0.01	−1.77
Collaboration outside national research agency		−0.32	0.41	−0.78	0.73	1.23	0.19	6.56
Betweenness		0.18	0.02	10.66	1.20	0.17	0.01	32.45
Betweenness (zero-> 1)		0.07	0.18	0.39	1.07	1.06	0.06	16.98
Gender (male->1)		0.05	0.07	0.68	1.05	0.07	0.02	3.24
Scientific age		0.46	0.25	1.87	1.58	−1.01	0.08	−12.46
PhD (yes->1)		2.09	0.06	33.58	8.08	1.12	0.02	45.25
Field: engineering sciences and technologies		−0.66	0.26	−2.56	0.52	−0.14	0.11	−1.23
Field: medical sciences		0.65	0.29	2.26	1.92	−0.15	0.12	−1.23
Field: biotechnical sciences		−0.62	0.31	−1.99	0.54	−0.44	0.14	−3.26
Field: social sciences		0.56	0.27	2.08	1.75	0.11	0.12	0.99
Field: humanities		3.19	0.32	9.90	24.29	1.11	0.11	9.79
Adjusted (pseudo) R²					0.486			0.491
Variance explained	Discipline level				0.936			0.896
	Residuals				0.287			0.412

Table 3

Summary table of estimated parameters for frequency and severity in the SE model. The left side presents the results of the logistic regression on the complete sample (frequency). The right side presents regression results of the sub-sample (variable SE > 0) (severity). The reference category for the research field is ‘Natural sciences and mathematics’. Bold values are significant at 0.05 level.

		Frequency				Severity
		est.	s.e.	z value	odds r.	est.	s.e.	t value
(Intercept)		−1.33	3.59	−0.37	0.26	23.21	2.04	11.36
Fragmentation of funds		−0.04	0.01	−3.10	0.96	−0.02	0.01	−2.11
Collaboration outside national research agency		2.78	0.37	7.48	16.12	2.12	0.25	8.52
Betweenness		0.20	0.01	16.49	1.22	0.11	0.01	16.46
Betweenness (zero-> 1)		0.69	0.14	4.90	1.99	0.91	0.08	11.34
Gender (male-> 1)		−0.06	0.05	−1.11	0.94	0.12	0.03	4.14
Scientific age		0.15	0.18	0.83	1.16	−0.92	0.10	−8.91
PhD (yes-> 1)		1.62	0.05	30.78	5.05	0.70	0.04	18.87
Field: engineering sciences and technologies		−0.53	0.24	−2.22	0.59	−0.08	0.13	−0.58
Field: medical sciences		0.13	0.27	0.51	1.14	−0.40	0.14	−2.80
Field: biotechnical sciences		−0.62	0.29	−2.13	0.54	−0.43	0.16	−2.71
Field: social sciences		−1.26	0.25	−5.00	0.28	−0.39	0.14	−2.78
Field: humanities		1.11	0.25	4.46	3.03	0.81	0.13	6.06
Adjusted (pseudo) R²					0.419			0.350
Variance explained	Discipline level				0.844			0.910
	Residuals				0.208			0.191

3.4.1 The first scientific publication

In the first part of the analysis of SP, we model which factors contribute to researcher's first scientific publication (see left part in Table 2). This model has the smallest number of significant parameters which means that whether or not a researcher publishes at all is affected by having a doctoral degree and by network betweenness. In other words, odds of publishing at least one scientific publication are 8.08 times higher for researchers who have PhD. Starting to publish is very important, especially for those intent on professional careers at academic institutions. Similarly, there is an obvious connection between a higher betweenness centrality based on the co-authorship network and the publication (nowadays typically co-authored).

The effects of the fragmentation of funds, the collaboration level and demographic characteristics of the researchers are not statistically significant.

As mentioned, the publication scoring differs between scientific fields. To control these differences, we added as control variables the field affiliation. The engineering sciences and technologies have a significantly low value of the estimated parameter (−0.66), which indicates a high number of researchers who have never made a scientific publication. The engineers are followed by researchers from the social sciences with a positive parameter (0.56), the medical sciences (0.65) and researchers from the humanities with a very high value of the parameter (3.19) (which is expected because of the loose rules to calculate the total score). These results can be explained by the high proportion of technical staff in disciplines belonging to engineering sciences and technologies. The parameter is not significant in the biotechnical sciences.

3.4.2 Who are the productive ones?

The analysis of SP focuses on quantified differences among researchers who do publish. The strongest effect on the SP comes from collaboration outside the SRA (1.23) (right part in Table 2). This means that those researchers who publish in co-authorship with foreign authors publish more than others. Another strong factor affecting SP is the betweenness parameter with a positive value (0.17), indicating that more productive authors are those who are well embedded in the national scientific co-authorship network. The strong effect of (zero) betweenness on high productivity is slightly intriguing, but makes sense when taking into account the fact that zero betweenness indicates researchers who publish alone or only publish with one co-author, as well as those who collaborate outside the network of Slovenian researchers. This shows that authors positioned on the border of the Slovenian collaboration network are also more productive.

The controlling variables show higher productivity of researchers with a PhD (1.12) and slightly higher productivity of male researchers compared to female researchers (0.07). Regarding age, there is a negative significant effect (−1.01), which indicates higher productivity of older researchers.

Estimated parameters in the research fields indicate that only being affiliated with the biotechnical sciences results in a significantly lower level of measured SP (−0.44) with regard to the natural sciences; conversely, affiliation with the humanities has a strong positive effect (1.11) on measured productivity. The high productivity of the traditionally undervalued researchers from the humanities can be attributed to the scoring correction for this field described in Section 3.2.

The distribution of funds has no significant effect on SP. This can be explained by the habilitation criteria of the universities where higher productivity matters. University scholars are highly motivated to be scientifically productive in order to be able to climb the steps of the university hierarchy, regardless of whether or not they have research funding.

3.4.3 Joining the elite

The third part of the analysis is based on the second model where ‘having at least one scientifically excellent publication’ is the dependant variable. First, the factors that drive researchers to publish at least one scientifically excellent publication are tested (see left part in Table 3).

The first difference between the models of SP and excellence is the significant negative estimated parameter value of the fragmentation of funds (−0.04). The negative value indicates that researchers from disciplines with a lower level of fund fragmentation (having a bigger proportion of funding from a single project or programme) are more likely to publish excellent publications than those from disciplines funded from various fragmented sources. In terms of odds, if number of researchers financed by one FTE drops for one researcher, odds for publishing at least one excellent publication rise for 4 per cent. This is the crucial difference in comparison to the results of SP, where the fragmentation level is not significant.

Concerning the results above, it is necessary to say that the Slovenian system of R&D public funding has remained practically unchanged over the last twenty years. This is despite the fact that simultaneously the institutional structure of R&D activity was continuously reorganised. Its main characteristic is the lack of a concentration of financial resources. The model of financing research groups in the public R&D sector has also remained relatively untouched in the period of the big reduction of funding for R&D at the national level over the last five years.

Although some studies which examine the role of grant size in research funding note that in several countries there is an increasing focus on the concentration of money for R&D activities (Bloch and Sørensen 2015), Slovenia as a small country with very limited R&D resources has not followed this pattern. We are encountering the following paradox: small countries are often accused of copying certain R&D policy models that have proven to be successful in big countries. In the case of financing R&D, just the opposite applies. Slovenia has so far failed to overcome the extreme fragmentation of R&D funding. One of the negative consequences of this situation is that those scientific disciplines most strongly subjected to the considerable fragmentation of research funds also achieve less SE. This is reflected in our research results. Thus, there is the danger that the trend of fragmented financing of R&D in Slovenia will continue relatively untouched, even though the smallness of the scientific community in Slovenia could be the best argument to fight against this phenomenon. As noticed by other studies, the restructuring processes which might lead to more effective models of R&D funding are far from being complete in most CEE countries (Lepori et al. 2009; Radosevic and Lepori 2009). Our analysis is based on the situation in Slovenia over a ten-year time period. As the Slovenian R&D system is not too different from those in other CEE countries (a general presupposition, however not empirically tested), our assumption is that very similar (negative) R&D funding trends in these countries can be anticipated.

The three parameter estimates used to model the effects of collaboration on publishing at least one excellent scientific publication are all positive and significant. The largest and also biggest in the model is the effect of collaboration outside the SRA (2.78). This means that each time when the ratio of researchers from abroad rises for one unit, odds of publishing at least one excellent publication rise sixteen times. This confirms that scientific isolation within the national borders is not good for SE. In fact, this was never the case in the history of science. Different forms of co-operation between scientists have long been an important element in the internationalisation of science. Notwithstanding, in the words of John Ziman ‘the traditional cosmopolitan individualism of science is rapidly being transformed into what might be described as transnational collectivism’ (Ziman 1994: 218). Science is now moving beyond national borders and becoming international.

Similar as when modelling SP, the Slovenian researchers who publish excellent publications are more likely to be well connected within a national scientific co-authorship network and have a statistically significant effect on betweenness (zero) (0.69) and on betweenness (0.20).

Among the effects of the controlling variables, only having a PhD has a positive significant effect (1.62), which means that those with a doctoral degree are more likely to publish at least one excellent scientific publication.

Regarding the differences among research fields, publishing scientifically excellent publications is least common in the social sciences (−1.26) followed by the biotechnical sciences (−0.62) and engineering sciences (−0.53). In the medical sciences, the parameter level is not significantly different from the one in the reference category of the natural sciences. The highest positive significant parameter estimate is again in the domain of the humanities (1.11), due to the specific scoring of excellence in comparison with other fields.

3.4.4 What makes an excellent scientist?

In the right part in Table 3, the effects of factors that contribute to the degree of SE are tested. In general, the most successful researchers are older (−0.92), male (0.21) researchers with a PhD (0.70), who often publish with authors from abroad (2.12). They are well embedded in the national co-authorship network (betweenness (zero) = 0.91 and betweenness 0.11). A greater number of excellent researchers work in an environment with less fragmented funding (−0.02). The results of this model are in accordance with other modelling segments.

An important conclusion is that there is a difference in the effect of the fragmentation of funding on SP and SE. Less fragmented funding helps in publishing the first excellent contribution, and also contributes to the excellence in general. In line with other models, the most excellent researchers are from the humanities (0.81), according to the reference field of the natural sciences and mathematics. Belonging to the scientific fields of the social (−0.39), medical (−0.40), and biotechnical (−0.43) sciences has negative effects.

4. Excellent Slovenian scientists about SE and indicators of excellence

As we mentioned earlier, analysing the SRA quantitative indicators of research performance in its research evaluation methodology and testing which policy factors have an impact on them, can provide us with a certain degree of confidence regarding their appropriateness for the Slovenian scientific system. Still, there are other factors and processes of research productivity and excellence that remain embedded in the scientific community and the researchers themselves, which are not readily captured by quantitative indicators. In order to shed some light on such often obscured features of research performance, we decided to complement our quantitative analysis of publication productivity and excellence indicators with interviews conducted with a small number of excellent Slovenian researchers.

The goal was to: (1) discover their views on SE and the appropriateness of excellence indicators, as well as possible additional field-specific elements of excellence not necessarily captured by the quantitative indicators; and (2) to inquire about their views on the importance of collaboration, specifically of co-authorships, in their research work.

We conducted the semi-structured interviews with six excellent Slovenian researchers between November 2015 and January 2016. We selected one representative from each of the six scientific fields in the Slovenian scientific community, using excellence as the measure of selection. For this purpose, we looked at the SRA quantitative indicators and qualitative factors such as public visibility, receiving prestigious national awards, being present in intellectual debates, etc. The interviewees were anonymised according to their scientific fields as natural and mathematical sciences, technical sciences, medical sciences, biotechnical sciences, social sciences, and the humanities. This is a small sample of individuals from specific disciplines, which in no way offer a general insight into the opinions and aspects of their respective fields, or even neighbouring disciplines. Nevertheless, they do offer some broader glimpses into how excellent scientists perceive excellence and what the indicators used to quantify research performance do not necessarily show and capture.

(1) We first inquired about their views on what makes up excellence in research and whether they see the quantitative indicators as a proper measure of excellence.

Natural says that excellence in his/her field is evident in whether or not someone is known abroad. This is reflected in being invited into European projects, as a guest lecturer, as a conference speaker, being able to exchange students with excellent institutions abroad, in short, being recognised as excellent among colleagues abroad. (S)he felt that educating an excellent researcher who can contribute further excellent work either in academia or the industry should also be seen as part of excellence.

Technical sees excellence primarily as arriving at original results, and achieving new discoveries before colleagues from abroad do. The results of this are publications in high-IF journals and invited plenary lectures. Regarding the quantitative indicators, (s)he sees them as a proper measure in themselves, although (s)he points to abuses and attempts to creatively circumvent the system. Technical further emphasises the need to (also) consider the actual content of publications when evaluating excellence, of experts evaluating the excellence of the content, which is not revealed quantitatively and, as (s)he says, numbers are easier to abuse and distort than content. Each element, such as acquiring funds from various projects and sources, educating new researchers, reveal some part of excellence, but do not capture the whole picture. (S)he sees the exchange and employment of former students at excellent institutions abroad as a major indicator, which is not taken into account.

Medical feels that excellence is demonstrated by publications in international IF journals, and that this is necessary despite the deviations that crop up. What is also important and should be seen as excellent is transferring and implementing new methods, establishing a new laboratory, and educating and training the member of a research group.

Biotechnical stated that excellence is what is recognised by others as excellent. In this sense, it is an internationally recognised result that can be either a publication in an excellent journal, an internationally recognised patent or product. Using top IF publications as an indicator of excellence is seen as appropriate, at least as long as the editors and reviewers are excellent experts. (S)he does, however, point to citation counts as being problematic, at least for their discipline, as most citations accrue to research that is more mainstream, but not to what is cutting edge and ground-breaking.

Social points out that it is difficult to measure excellence, but that it can be found in its results by measuring the quantity and quality of publications. As the Slovenian scientific space is too small to be able to establish some at least partially objective list of excellent journals, (s)he focuses on international journals. These are in the upper half of the SSCI, along with attendances at conferences considered excellent for his/her field. (S)he also noted that the various indicators and elements should be judged separately, not as an aggregate.

Humanities was generally critical of quantified attempts to measure excellence, stating that such criteria set up by the government and funding agencies have no connection with the standing and actual scientific content produced by a research or programme group. In his/her view, such measures are not really objective but are a reflection of interest groups and increasing bureaucratisation. The result of this is avoidance of personal responsibility, of actually writing a content evaluation of some work or some researcher and standing behind it with one's own name. Further, (s)he points to the problem of properly measuring some quantitative indicators in the humanities, such as citations, which are required for project and programme group leaders. As most citations in the humanities are made in monographs, these currently need to be collected manually by each individual.

In general, we can see that for most interviewees excellence is something that is broader than what is captured by the quantitative indicators used to measure SE. This pertains especially to peer recognition and visibility that extends beyond publications in prestigious journals. Practically all hold the view of excellence as excellent new research that is recognised by peers, preferably internationally, and this is reflected in invitations to projects, lectures and conferences, student and researcher exchanges. This is close to the fuzzy concept of SE embedded in the scientific community and determined through peer review (Sørensen et al. 2016). Nevertheless, most do see measuring publication excellence through publications in international IF journals as a reflection or indication of excellence despite some abuses and possibilities of gaming the system, although not sufficient by itself. An exception to this view is Humanities, which has field-specific problems due to publication differences with regard to other fields (and, as we mentioned previously, a more lenient scoring system compared to the other scientific fields). The content of the publications and research work is still emphasised as being crucially important in discerning excellence as both the IF of the journal and the citation count might not properly reflect breakthrough and cutting-edge research work.

(2) Second, we asked the excellent scientists about the importance of collaboration, specifically of co-authorship collaborations, in their research work. As can already be seen from the first part, many interviewees point out the importance of international collaboration through projects, researcher and student exchanges, as well as other types of joint research endeavours, as both prerequisites and reflections of excellence.

Humanities mentioned that both international and interdisciplinary cooperation are important since most research problems require different aspects and viewpoints to be addressed, but this pertains mainly to joint projects and research while co-authorships are rare as far as journal articles are concerned. Social said that multidisciplinary connections and cooperation are crucial both nationally and internationally, and usually result in co-authorships. Technical says that they almost always cooperate with colleagues from abroad as well as with Slovenian colleagues from other disciplines, which results in co-authorship publications. Natural stated they often engage in interdisciplinary research, which leads to co-authorship publications, both national and international. Medical mentioned the necessity of multidisciplinary collaboration, the results of which are co-authorships with both domestic and foreign co-authors. Biotechnical lists interdisciplinary cooperation, domestic, and international, which results in co-authored publications.

Overall, we can see that practically all of the interviewees consider both interdisciplinary and international cooperation to be extremely important for their work. What is evident from the broader context of the interviews is that co-authorship publications are a result of such collaboration and can be seen as a reflection or indication of excellence. Of course, again, this is a small sample that cannot offer general conclusions or insights about the fields or disciplines, but does hint at a connection between cooperation, excellence and co-authored publications.

5. Conclusion

We attempt to explain and analyse two dimensions of scientific performance, that is SP and SE, in the context of the Slovenian scientific community and its research performance. Over the past decade, both dimensions have become increasingly concretised in various quantitative bibliometric indicators which are used in decisions to distribute R&D funds. The R&D policy trend to measure SP and SE through quantitative bibliometric indicators is a recent, world-wide phenomenon. Such indicators are important especially for small countries with small scientific communities since they can provide a measure of accountability and transparency of both scientists and policy decision-makers. Following the political changes in the 1990s, Slovenia also became part of the global trend of using bibliometric indicators as instruments for R&D evaluation procedures by research agencies. In the Slovenian context, SP is measured through the complete production of scientific and non-scientific publications. While SE pertains only to articles published in top-ranking international journals and to books published by leading international publishing houses (with some exceptions in the scientific field of the humanities). In Slovenia, both are defined in the evaluation methodology of the SRA, and both can be affected by other science policy mechanisms such as funding and the promotion of co-authorship collaboration practices.

In the quantitative part of our analysis we used a bibliometric analysis of both dimensions in order to evaluate the research performance of Slovenian researchers. We further investigated the impact of funding and collaboration (Causality in the model could be questioned since we assume that fragmentation affects scientific performance. It is possible that scientific disciplines which achieve lower scientific excellence could be subjected to higher funds fragmentation.). We confirmed that in Slovenia the fragmentation of funding and the collaboration (as observed through the co-authorship of publications) of scientists within and beyond national borders both have a very strong positive effect on SE. But only the collaboration of scientists has a strong positive effect on SP. Not surprisingly, the results of our analysis are in accordance with the conclusions of other R&D evaluation studies which investigated the contextual factors of research and technological performance in countries with small scientific communities (e.g. Wagner and Leydesdorff 2005; Leydesdorff and Wagner 2008; Oliveira and Carvalho 2011).

Still, as some of the criticisms of bibliometric indicators show, they cannot provide the whole picture of all the mechanisms and processes underlying productivity and excellence, although they can serve as reflections of such processes. In order to gain a more holistic picture, we also conducted interviews with a small group of excellent Slovenian researchers, representatives of all main scientific fields. Given their status as excellent researchers, these scientists should have at least some implicit ideas of what makes up excellence and which factors might affect it since they are also the ones primarily affected by the increasing use of bibliometric indicators in evaluating research performance. As the interviews show, most of them caution that excellence is something that is broader than what is captured by the quantitative indicators used to measure SE alone. While quantitative indicators can serve as reflections and results of excellence, they are not enough by themselves. What is also important in order to promote truly excellent and breakthrough research is taking qualitative stock of the content of the research and publications. Regarding the factor of inter-institutional and international cooperation, practically all the interviewed scientists recognise the importance of this element, which is reflected in co-authored publications.

Ultimately, what is important is a proper combination of quantitative and qualitative evaluation instruments, that is, peer review conducted by knowledgeable and unbiased reviewers who know how to properly use the results of complex quantitative indicators. On the side of national and transnational science policies, quantitative bibliometric indicators are becoming an integral and indispensable part of R&D performance assessment. But policymakers still need to take into account that there are additional factors and processes that are not necessarily captured by the indicators, but are nevertheless indispensable for ensuring SP and excellence. We need to keep in mind that even in the era of big science, where globally the processes of internationalisation and rational distribution of resources are very important for promoting SP and SE, such processes are complex and require a holistic approach for proper understanding and management.

Conflict of Interest: None declared.

Acknowledgement

This article was written with the support of the Slovenian Research Agency funded project, Cooperation networks in Slovene Science, grant number J5 5537.