- Split View
-
Views
-
Cite
Cite
Mark W Neff, Publication incentives undermine the utility of science: Ecological research in Mexico, Science and Public Policy, Volume 45, Issue 2, April 2018, Pages 191–201, https://doi.org/10.1093/scipol/scx054
- Share Icon Share
Abstract
Governmental spending on science is usually justified by claims that the resulting research will yield benefits for the sponsoring nation. I present policy-analytic and ethnographic research—based on 30 hour-long interviews—of the Mexican ecological research community to explore the structural influence of publication incentives on research content and its relevance to national needs. During a financial crisis in the 1980s, Mexico created a national publication incentive system, the Sistema Nacional de Investigadores, to identify and reward scientists producing the most and the most-cited research as defined by dominant international scientific norms at the time. The system has increased productivity but in the process has undermined that country’s ability to benefit from its ecological research by surrendering priority setting to the editorial preferences of journals that are linguistically and financially unavailable to potential domestic users. The Mexican experience has implications for institutions worldwide that utilize quantitative productivity measures in researcher evaluation.
1. Introduction
Societal support for science stems from the expectation that science benefits society financially, by contributing to technological innovations, and by informing decisions (Sarewitz 2004). These expectations were given prominent voice by Vannevar Bush in his influential 1945 essay Science—The Endless Frontier, which portrayed scientific advance as a pre-condition for societal improvement (Bush, 1945; Stokes, 1997). Bush’s argument came in an essay, solicited by then-US President Franklin D. Roosevelt, proposing that the USA needed to maintain its scientific capacity upon the end of the Second World War. Each country—in his estimation—has to do its own basic scientific research in order to advance its economy; ‘A Nation which depends on others for its new basic scientific knowledge will be slow in its industrial progress and weak in its competitive position in world trade, regardless of its mechanical skill’ (Bush, 1945). This aspect of his thinking resonated widely, as is reflected in the current trend for international organizations, nations, and research institutions within those nations to use the size of budgets and counts of publications as metrics of the adequacy of their scientific enterprises (e.g. Lemarchand, 2010).
Bush took the opportunity to argue not just that the nation would benefit from substantial government funding, but also to assert that scientific knowledge would yield the greatest societal benefit if ‘scientists are free to pursue the truth wherever it may lead’ (Bush, 1945). Despite the countless technological innovations and scientific breakthroughs that observers today trace to wartime research (e.g. Sarewitz, 2016), Bush saw government efforts to steer scientists toward knowledge needs during Second World War as ‘rigid controls’ that minimized the contributions that science could make.
Michael Polanyi (1962) further articulated the argument for scientific autonomy when he wrote that scientists themselves are best positioned to make decisions regarding governance of their own efforts. Any centralized attempt to influence the choices of individual scientists or direct them toward societally important projects would ‘bring the progress of science virtually to a standstill’ (Polanyi, 1962). Collectively, the ‘independent initiatives’ of individual researchers, ‘adjusting themselves at every successive stage to the situation created by all the others who are acting likewise’, creates—he argued—a scientific field that is coordinated ‘as if by “an invisible hand” towards the joint discovery of a hidden system of things’ (Polanyi, 1962). And he wrote that the three criteria scientists use in evaluating the merit of research are plausibility, scientific merit (including accuracy, systematic importance, and intrinsic interest), and originality. To him, these are the only appropriate considerations when evaluating research subjects and methods, and scientists alone should be responsible for these assessments.
Taken together, Bush’s and Polanyi’s arguments constitute key components of what has come to be known as the linear or loading-dock model of the science-policy interface (Stokes, 1997; Cash et al., 2006; Godin, 2006). The basic idea of the linear model is that self-governing scientists, pursuing their own interests and steered by their shared community, generate a pool of robust and disinterested basic knowledge. Other actors then draw from that pool of knowledge for subsequent use in technological development or decision-making. By the logic of this model, any increase in financial resources available to basic science yields a larger pool of knowledge and therefore more eventual societal benefit. Successful science policy under this model is thus assumed to be anything that increases publication counts (the size of the pool) or citations (a severely limited but measureable metric of knowledge use; see Moustafa (2016)). The logic of the linear model has in recent decades come to be taken for granted within many scientific and policy communities (Sarewitz et al., 2004; Pielke, 2007).
The linear model is appealing to scientists because it justifies generous funding with minimal external oversight. And it is a tempting myth for actors both internal and external to science in that it appears to maintain a tidy distinction between scientific activity and eventual use of knowledge, seeming to buffer scientific processes from the realm of values and politics (Douglas, 2009). This asserted separation of values entailed in knowledge production from social and political ones is part of the reason why actors in modern liberal democracies frequently invoke science in advocating their preferred policies (Ezrahi, 1990; Barke, 2003; Douglas, 2009).
Despite its intuitive appeal to scientists and non-scientists alike, however, the linear model has been dismissed as empirically uninformed wishful thinking for decades by scholars who study knowledge creation and uptake. Producing more science does not inherently yield better outcomes. Just as science can contribute meaningful societal benefit, it can also amplify economic inequalities (Sarewitz et al., 2004; Woodhouse and Sarewitz, 2007), cause or contribute to environmental challenges (Woodhouse, 2006), and exacerbate policy controversies rather than dispel them (Collingridge and Reeve, 1986; Sarewitz, 2004).
One substantial shortcoming with the linear model to which many scientists can relate is that it is far from assured that a knowledge user will make use of relevant research when it does exist. For a user to make use of knowledge requires that s/he find the relevant scientific information from the millions of peer-reviewed and published articles and books, recognize its utility to their current situation, deem it to be credible, salient, and legitimate, and know how to make use of it. An emerging literature suggests that for science to be most effective at contributing solutions to society’s problems, scientists must interact with potential knowledge users in an ongoing and iterative fashion, starting with the preliminary task of defining research questions and selecting methods for analysis, continuing throughout the research process, and including communication and possible utilization of the results (Cash et al., 2003; Cash et al., 2006; Roux et al., 2006; Sarewitz and Pielke, 2007; SPARC, 2010; Cockburn et al., 2016; Sarewitz, 2016).
Vannever Bush and many since, have advocated for granting scientists substantial autonomy on the premise that we cannot know from the outset what outcomes—though assumed to be beneficial—will eventually result from research. Although history bears out his assertion that societal benefit can result from serendipity, it is both a logical fallacy and historically uninformed to assume therefore that serendipitous outcomes cannot also derive from research priorities accountable to broader democratic processes (Sarewitz, 1996; Stokes, 1997; Sarewitz, 2016).
An understanding that societal benefit does not automatically flow from scientific work suggests that informed science policy requires reflective awareness of the social and cultural factors at work within science that help to shape and define scientific research portfolios and the relationships between producers and users of knowledge. Publication norms and policies constitute one example: Vessuri et al. (2014) argue that the desire to publish in a ‘core set’ of international journals, defined predominantly by the impact factors (IFs) of those journals, so dominates the cultures of science in Latin America that it may affect the form and content of the research. Science policies in the region must be informed by the dynamics of the publication industry, including the activities of scientists in light of publishers’ and journals’ editorial activities:
… who controls [the journals]? To what end? Journals play a leading role in creating symbolic value, but how does it relate to the revenue-making motive of most publishers? Also, what scientific questions are being slighted or even ignored with the present system of journal competition? Is it possible that, as result of this regime, scientists from the ‘periphery’ actually contribute more to problems affecting mostly rich countries (a kind of foreign aid in reverse) rather than their own? (Vessuri et al., 2014)
Mexico constitutes an ideal case study to evaluate these hypotheses because national policies there directly incentivize scientists to pursue publication in these ‘core’ international journals. I use policy-analytic and ethnographic methods to document Mexico’s researcher evaluation schemes as science policies—built around the norms of scientific self-governance—and gauge their impacts on ecological research portfolios and practices in that country. For science to contribute meaningfully to society requires that the policies and institutions governing science be designed to encourage or at least be compatible with that outcome. Current researcher evaluation schemes in Mexico, though well-intentioned and built upon the dominant ideas of what constitutes ‘good’ science, undermine the ability of ecologists in that country to contribute toward the knowledge needs of their country.
Mexico’s researcher evaluation scheme, known as the Sistema Nacional de Investigadores (SNI; ‘National System of Investigators’), offers substantial financial rewards to scientists with the highest productivity levels. In practice, for most ecologists, productivity is equated with the number of articles published in Web of Science (WoS)-listed journals—with emphasis on those with higher IFs—and citations from those same journals. Most universities and research institutes have additional reward systems that mirror (to differing degrees) the national SNI system, as do many states in Mexico. The net result is that it is possible to double or triple your salary as an ecologist if you publish sufficient articles in journals with high IFs and accumulate adequate citations in those same journals.
I first introduce the historical motivations behind the creation of the SNI. Second, I describe the suite of journal and article databases that SNI relies upon for evaluation and introduce several of the well-trodden mathematical critiques of those products and their application in evaluations of individual researchers. Third, I present an analysis of how research evaluation systems such as the SNI impact the content and process of ecological work. The results derive from analysis of written policies as well as of 30 approximately hour-long semi-structured interviews that I conducted with Mexican ecologists, research managers, and government ministry users of scientific knowledge from 2013–6. In the tradition of qualitative social science, I triangulate between different sources and formats of data, including interview transcripts, policy documents, and published academic research, to explore the impacts of publication incentive schemes on the process, content, and usability of Mexican ecological research (Miles and Huberman, 1994; Bernard, 2006). The narratives and specific examples differ from scientist to scientist and institution to institution, but the analysis focuses on themes and experiences shared broadly by the scientists with whom I spoke.
This work is part of an overall project examining research incentives as science policies in five countries: The USA, Canada, Mexico, Brazil, and Peru. The project is intended to shine a critical lens evenly upon all of these national science systems. In this article, I highlight both the intended and unintended impacts of research policies in Mexico in the spirit of helping to inform subsequent policy dialogues. The dynamics I describe are not exclusive to the Mexican scientific system; in fact, related problems permeate much of modern science regardless of national setting. Mexico is not the only country to use simplistic proxy measures of researcher contributions but it provides an excellent opportunity to evaluate the effects of that form of researcher evaluation because its policies are well-articulated and apply nation-wide.
2. Background
2.1 Historical genesis of Mexico’s SNI
In the decades leading up to the 1980s, oil had become a significant source of government revenue in Mexico; when the price of oil plunged in 1981 and 1982, the national economy took a big hit. Mexico at the time was the largest debtor nation in the world, and without the oil revenue on which it depended, it could no longer meet its debt obligations to international commercial banks (Hellman, 1997). The World Bank and International Monetary Fund intervened, but demanded substantial structural adjustment of the Mexican economy. By 1988 interest payments on international loans came to constitute 57 per cent of total government expenditures, and its only option was to devalue its currency. Annual inflation during this time climbed as high as 140 per cent (Bello, 2008). Suddenly, academic salaries that were once sufficient for a comfortable lifestyle became almost worthless. Researchers took one of two responses: some fled to countries that paid higher salaries, and others treated their secure lifetime positions as sinecures and took on second or third jobs to make ends meet. Both responses had the effect of undermining scientific work in the country, and together they represented a crisis for Mexican science.
In 1983, as this crisis began to unfold, the Academia de la Investigación Científica (the National Academy of Science) worked with the Minister of Education to convince the President that Mexico needed a policy capable of stemming the brain drain (Heras, 2005). The proposed solution was to identify the most productive scientists and reward them with salary bonuses. This was the genesis of the Sistema Nacional de Investigadores (National System of Investigators), known to Mexican scientists by its acronym SNI. Under SNI, scientists were asked to report their productivity, and those scientists who were producing the most were awarded additional pay; those who were not producing much were left with their base salaries which were greatly devalued by inflation. The dual intent was to encourage researcher productivity and prevent a brain drain from the country; in both regards, the system worked and many— but not all— of the early proponents of the system still see it as a dramatic success.
This is a point worth emphasizing: Mexican science was at risk of severe damage, and if we accept science to be a worthwhile activity, we must recognize that the SNI played a crucial role in maintaining that activity. Indeed, by the problematically simplistic measure of publication counts in ‘recognized’ journals—the metric by which much of the global scientific community judges itself—the country did quite well. Individual researcher productivity increased across the board (Gonzalez-Brambila and Veloso, 2007), and between 1973 and 2006 total Mexican scientific productivity was the second-highest in Latin America, behind only Brazil (Lemarchand, 2010). Mexican science would surely have suffered immeasurably during to the debt crisis in the absence of SNI. But, the system was not and is not perfect.
2.2 Research evaluation under the SNI
The SNI ranks sufficiently productive Mexican researchers into categories according to their productivity (Candidate, and Levels I, II, III, and emeritus) with non-taxable salary bonuses that increase at each level. Researchers meeting minimum standards are accepted into the system as candidates, and are re-evaluated periodically at intervals depending on their ranking. They can climb the rankings and eventually receive emeritus-status with a permanent lifetime salary when they have achieved sufficient international recognition. The system relies heavily on publication databases maintained until recently by Thompson Reuters, including the WoS, the Science Citation Index (SCI), and Journal Citation Reports (JCR). In 2016, corporate investors purchased the Thompson Reuters Intellectual Property and Science business and incorporated those products under the name Clarivate Analytics (http://clarivate.com/news/ip-and-science-launched-as-independent-company/). Reflecting the colloquial terminology of Mexican scientists, I refer to these database products collectively as ISI.
Because SNI rewards the most ‘productive’ scientists (as measured largely by publication count in and citations from ISI-listed journals) and because the financial rewards for such productivity can be significant, individual institutions are under little pressure to increase base salaries (Castaños-Lomnitz, 2003). Doing so would both impact their budgets and decrease the effectiveness of the productivity incentives. And the institutions themselves receive more funding from the main federal funder of science, the Consejo Nacional de Ciencia y Tecnología (National Council of Science and Technology, referred to by the acronym CONACyT), when their scientists are more productive. Thus, individual institutions often bring their own publication incentives into alignment with the national system.
The goals of the SNI are laudable, and the evaluation is built upon peer review—scientists evaluating other scientists in the model of self-governance—which most researchers hold up as a gold standard of scientific research and the driving force behind scientific self-governance. Seen through the lens of the linear model, it is a productive science policy in that it focuses on increasing the size of the available pool of knowledge, as evaluated by standards familiar to the international scientific community. But the merits of the evaluation scheme diminish upon further inspection. When first created, the system had only three committees to evaluate the portfolios of all researchers in all of the varied disciplines within the country, including ethnography, medical science, astrophysics, and everything in between. The committee that evaluated ecologists evaluated not just the many subfields of ecology, but also the full spectrum of biological, biomedical, agricultural, and chemical sciences. It was quite likely under this system that the committee evaluating an ecologist would include no ecologists.
The number of committees has since increased to seven (Auditoria Superior de la Federación, 2009) , but at least three basic problems remain: (1) Scientists are evaluated by committees that may not contain a single member with similar expertise; (2) Publication cultures and rates can vary dramatically between disciplines, and indeed between even members of a single discipline that utilize different methods; and (3) Each of these committees is made up of fourteen senior researchers who themselves must maintain productivity while evaluating the productivity of 500 or more scientists over the previous 3 to 5-year period (Ricker et al. 2009). To complete that number of reviews of scientists in unfamiliar fields, alongside their other work responsibilities (including maintaining the productivity that will allow them to stay within the system themselves), critics argue that reviewers have no choice but to conduct cursory analysis of quantifiable indicators of quality: how many publications does the applicant have, what are the IFs of the journals in which they appear, and how many citations have those publications received (Ricker et al. 2009)?
The rules for evaluating researchers vary by committee and can be altered by the committees themselves. Researchers need SNI-3 status to be eligible for membership on the evaluation committee, and current committee members elect members as vacancies occur. This process ensures that those disciplines with faster publication rates remain strongly represented on committees that most strongly value publication counts. Agency to bring about change is strongest for those who have succeeded under the current system; scientists whose research yields fewer publications in ‘prestigious’ journals—regardless of their importance for Mexico—have little influence over future evaluation rules.
Each committee promulgates rules that apply to all scientists in their domain, without differentiating by disciplines, methods, or institutions. Grounding researcher evaluation policies in disciplinary averages of publication and citation counts is problematic even within a single scientific field, but it is even more-so when expectations rooted in one discipline are applied to others. It is well documented that disciplines differ in terms of project completion times, citation practices, and other discipline-specific cultural and logistical factors (Radicchi et al. 2008); these differences are unrelated to the utility or importance of the knowledge produced. The committee that evaluates many ecologists, ‘Area II: Biology and Chemistry’, requires a specific number of publications and citations all researchers under its jurisdiction must have for admission to each level of SNI and requires that those publications and citations must be from within the ISI SCI(CONACyT n.d.). As this article documents, the policies serve as blunt instruments that incentivize ecological scientists to pursue projects that yield frequent publications to the expense of research projects and methods that—regardless of their potential utility to knowledge users in Mexico or to science writ large—may be slower, result in fewer publications and citations, or not be as appealing to editorial boards of journals with adequate IFs, most of which are based abroad.
Defenders of the system maintain that SNI evaluation is more thorough and nuanced than counts of publications and citations in high IF or ISI-listed journals (Williams and Aluja, 2010), and indeed several metrics of researcher impact beyond publication and citation data (e.g. formation of research groups) are included among Area II criteria (CONACyT, n.d. section 2.4) and some area committees do award points for publications appearing in a list of journals maintained by CONACyT itself. But the number of publications, the IF of the journals in which they prefer, and counts of citations accumulated from within ISI-listed journals feature prominently in the formal evaluation criteria (CONACyT, n.d., section 2.2; Ricker et al., 2010) and they are central in ecological researchers’ perceptions of the SNI. Even if the above defenders of SNI are correct that the actual evaluations are nuanced and not overly dependent on simplistic quantitative measures of scientific impact, the interviews suggest that ecologists believe the quantitative measures to be central to SNI evaluations. They are acting accordingly.
2.3 WoS and the SCI
Mexico’s SNI Area II researcher evaluation criteria—and to differing degrees those in other Area Committees—are built around WoS products (JCR, SCI, and Journal IF), collectively termed ‘ISI’ by the community. Publication in ISI-indexed journals yields rewards far greater than publication in non-indexed journals. Researchers earn credit also for the number of citations they receive from articles in indexed journals; those coming from other journals do not count. The ISI databases, however, are not comprehensive in their coverage, nor are they meant to be. The modern SCI is built upon a product from the late 1950s, the Genetics Citation Index, that Eugene Garfield created to curate the English-language literature in that field. He and others expanded upon that core set of journals by creating the IF, which identified the journals most frequently cited by authors in his genetics index. These were included in an expanded database, the SCI, which has grown from the seed of the original Genetics Citation Index (Garfield, 2006).
Garfield envisioned his indices as curations of the scientific literature, highlighting the best journals in covered fields (Garfield, 1990; Garfield, 2006;). There are, however, problems in ISI’s implementation of these goals. Current inclusion criteria are only vaguely described publically, but depend heavily on the number of citations from currently listed journals (Garfield, 1990; Laborde, 2011) despite the lack of any consistent relationship between citation counts and the actual contribution of the article to science or society (Bozeman, 2003; Bozeman and Sarewitz, 2011; Moustafa, 2016). The characteristics of the journals that were included initially—including language and discipline—thereby became systematic biases determining the fates of other journals seeking inclusion. The databases until recently contained very few if any Spanish- or Portuguese-language journals in any field, so researchers writing in those languages are rarely cited in listed journals. This constitutes a self-reinforcing and circular bias: the lack of citations to Latin American Journals in indexed journals becomes a justification for not including Latin American journals in future years (see also Archambault and Larivière , 2009). And, because scientists are rewarded for publishing in indexed journals, the language bias in the indices virtually guarantees that Spanish- and Portuguese-language journals do not receive as many high-quality submissions since would-be authors first seek publication in ISI-indexed journals (Laborde, 2009; Fischman et al., 2010; González-Alcaide et al., 2012; Vessuri et al., 2014).
Pressure from competing indices, which had better coverage of Latin American Journals, led Thompson Scientific (current owner of this suite of products) in 2008 to suddenly reverse recent decisions that had excluded a number of Spanish- and Portuguese-language journals. These business decisions more than doubled the number of Latin American journals in ISI products, but brought the total to a mere 242 (Alperin, 2014). Nothing had changed about the editorial policies of those journals, nor about the citations those journals had received; rather, to all appearances, Thompson Reuters made a business decision based on external competition (Laborde, 2011). The inclusion of these journals moderately improved ISI coverage; for contrast, Latindex, the most comprehensive index of the region’s journals, lists 5,408 Latin American journals ‘of academic interest’ meeting a set of minimum criteria and known to be actively publishing (Alperin, 2014). And one recent estimate found that 80 per cent of Mexico’s research output from 2005–11 was in national journals (López Castañares, 2013), most of which have no representation in WoS. Nevertheless, SNI incentives are built around WoS products.
In 2014, Thompson Reuters made the Scientific Electronic Library Online, an open-access database of predominantly Latin American journals, available through its Web interface (Packer, 2014). These actions to include more Latin American journals in its resources benefit researchers working in those regions, but it also underscores a key point about ISI’s indices: they are commercial products designed with a profit motive. The SCI comprises what amounts to an arbitrary subset of research journals, curated by a company seeking financial profit. Admission to the ISI SCI is determined by proprietary criteria that are demonstrably inconsistently applied. Any alignment between decisions rooted in profit maximization of corporate owners of the WoS and related products and the knowledge needs of the countries financing the science that yields ISI-listed publications is coincidental, not intentional or inherent.
2.5 The journal IF
Because the Journal IF plays a central role directly in researcher evaluation in Mexico and is also used to select journals for inclusion in the SCI, the design of the statistic merits evaluation. The IF is a mechanically objective yet systematically biased measure of the scientific merits of academic journals—the purpose for which it was created—and is a mathematically inappropriate metric for assessing the merits of individual articles published in them.
Journals can—and do—manipulate their IFs in several ways completely unrelated to scientific quality (Archambault and Larivière, 2009). For example, journals can manipulate the ratio of citable to non-citable items by including news items such as those that fill half the pages of Science and Nature. These news pieces are frequently cited and are thus included in the numerator of the calculation, but the IF calculation treats them as non-citable and thus they are not included in the denominator (c.f. Garfield, 2006). Another way journals can boost their IFs is to publish reviews, which on average are more heavily cited than original research (Garfield, 1990; Moustafa, 2015). A third way is to require authors publishing in that journal to cite articles in that journal; the practice of asking authors to list additional articles in that journal as recommended further reading is probably less effective, but serves the same purpose of steering more potential authors to content in a journal.
Since Thompson Reuters (and presumably the spun-off current owner, Clarivate Analytics) also tracks ‘auto-citations’, or citations from a journal to other articles in that journal, journals have had to develop more elaborate schemes to boost their rankings. Several Brazilian journals, for example, were recently suspended from the JCR for manipulating their IFs in a complex citation ring wherein they conspired to publish reviews that featured intentional citations to co-conspirator journals (Van Noorden, 2013). That editors would go through the trouble of intentionally boosting their citation frequencies, and do so in such a way as to disguise their efforts from Thompson Reuters, indicates how seriously journals take their IFs. To the extent that the scientific community agrees that good science is published in high IF journals, journals that fail to compete on that metric will cease to attract high-quality submissions.
Other flaws exist as well. Garfield created the Citation Index and related products for a specific commercial audience: the early customers of Current Contents, dominated by genetics, molecular biology and biochemistry (Garfield, 2006; Laborde, 2011). Because it suited the citation patterns of those fields at that time, Garfield elected to calculate a 2-year IF, rather than some other time-period. Disciplines, as argued earlier, vary significantly in terms of how long after publication appears it takes before citations to that publication peaks, and the 2-year window is a poor indicator of the total number of citations that articles in many disciplines receive (Archambault and Larivière, 2009; Vanclay, 2009).
The IF is an average of citations received by all articles in a given journal, but its use to evaluate the merits of individual articles in those journals is mathematically unjustified because the calculation says nothing about the distributions of citations to individual articles. Testing how well IF predicts citation to a given article in their field of mathematics, Adler et al. (2009) found that 32 per cent of randomly selected articles in a journal with a 0.43 IF had more citations than those in a journal with a 2.63 IF. The latter has an IF six-times higher than the former, yet articles in the former are cited more frequently than those in the latter nearly one-third of the time (see also Laborde, 2011).
These critiques are just a first-level overview of the technical problems of using WoS as the basis for quantifying individual researcher productivity. Based upon these and related issues, a number of prominent scientists and organizations have written about the need to move away from IF as a metric of article and author contributions (The PLoS Medicine Editors, 2006; Aitkenhead, 2013; Alberts, 2013; Pulverer, 2013; Sample, 2013; Schekman, 2013). The critiques in this article, however, stem not just from the technical inadequacy of the IF, but also from a concern about the steering influences of evaluation systems based on productivity and citation measures and their impacts on society’s ability to benefit from science.
Scholars working within the context of Latin America have begun to document that policies and cultural norms built around journal IFs are adversely affecting research there. Vessuri et al. argue, for example, that science in the region has become a competitive ‘quest for excellence’, as defined by primary authorship in a ‘core’ set of journals with high IFs (Vessuri, 2014. The dynamic is detrimental in a number of ways. Among these authors’ critiques are the ways in which it deprives local and regional journals of content, elevates a few scientists to international prestige while undervaluing the bulk of the scientific labor force essential to their work, penalizes risky and novel research, and places evaluation of quality and scientific importance ‘in the hands of an international oligarchy made up of publishers and large scientific societies’ (Vessuri, 2014). Prestigious journals’ editorial policies and decisions thus become an uncoordinated yet powerful force in shaping global research agendas and practices; researchers seeking ‘prestige’ must respond to priorities as defined by scientists and journals in the world’s wealthy countries. This manuscript seeks to document, specifically, how scientific norms and formal policies built around publication ‘prestige’ affect the content and usability of ecological research in Mexico.
3. The impact of the SNI on Mexican ecology
‘The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor’ (Campbell, 1979; Campbell, 2011).
Mexico adopted ISI-based metrics to evaluate scientists’ productivity based on an acceptance of the linear model of the science–society relationship and its constituent logic that scientific self-governance is optimal. Articulating his vision of how scientific self-governance works, Polanyi argued that although an individual's substantial depth in one area of science necessarily means more superficial knowledge of other areas, scientists are able to judge the merits of projects not only in their fields but also in adjacent areas (Polanyi, 1962). The designers of SNI and contemporary advocates thereof assert that the Area Committees can adequately evaluate the range of disciplines they cover via the metrics they select and administer. The IF of a journal in which an article is published and the quantity of citations from indexed journals that an article receives is, they argue, the currency of science: Good science is published in good journals and receives abundant citations. For my purposes, I am not interested whether SNI evaluation criteria are ‘objectively’ employed, though that is a widespread concern among Mexican scientists. Rather, my concern is the systematic steering influence built into an evaluation mechanism that uses publication and citation counts as easily-quantified-but-inappropriate proxies for societal contribution.
Many of the scientists with whom I spoke were initially in favor of the quantitative rationality implied by the SNI evaluation system. I spoke separately with several scientists who were instrumental in building the incentive programs at their institutions (systems based in part upon SNI rank), but who have since become quite skeptical of SNI because, as one articulated, the incentive systems push scientists to ‘focus on subjects that are easy to resolve’ rather than those that are most important. Easy to resolve, for various participants, meant research: taking place in relatively well-understood ecosystems; involving questions that were not prone to disruption by weather, climate, or other unexpected ecological dynamics; not dependent on long-term continuity of funding; addressable via analysis of existing data; and on topics that were appropriate for relatively uninitiated graduate students, who shared co-authorship with and thus benefited their advisors. Participants reported that salary bonus systems from SNI and their institutions penalized risky and novel research, projects that span multiple field seasons, basic biological surveys, and—in general—anything that was slow or not easily relatable to topics trendy in internationally recognized journals. Several noted that the most heavily cited ecological research topics rely on expensive and inaccessible equipment, and thus the best they could do was collaborate with colleagues at wealthier institutions on their projects.
In particular, participants saw success under SNI and institution-level publication norms as incompatible with use-inspired and participatory research, approaches that they felt to be more morally appropriate and to yield more robust knowledge. The desire to do research that is important to the region and to communities is strong among many of the ecologists with whom I spoke, and indeed several prominent ecological research centers in Mexico were founded with that goal in mind. Some remote communities with whom my participants worked perceived outsiders as a threat, given colonial history and the more recent phenomenon of international scientists and companies taking local biological samples and profiting from them. Scientists reported that working with and even among these communities required months of relationship building, time and effort that is not rewarded under current evaluation schemes.
There was a palpable frustration among many scientists—including those at institutions that were founded with the stated purpose of working with agriculturalists to advance sustainable development—that success by SNI standards has come at the expense of societal impact. Many of these scientists very much value establishing trust-based relationships and working collaboratively with knowledge users in order to address the knowledge needs of their regions. Several of them chose to work in those institutional settings explicitly because their personal goals matched the institutions’ stated missions; some in fact chose to work in Mexico specifically because they believed that Mexican scientific cultures valued that approach.
One scientist whose work focuses on evolution of agricultural crops typified this frustration. He conducts his research in collaboration with and in service of communities of small-holder farms growing native varieties. His work, he says, is typically not publishable in the highest IF journals because the emphasis there is on research that leads to commercializable crops because of the larger scientific audience that research enjoys internationally. Within his personal research portfolio, he finds himself balancing the work that he values based upon his commitment to the farming community with that which is publishable in high-IF journals because the base salaries are insufficient without the SNI bonuses. The base salary, he said, ‘is basically calculated so that you need [the SNI bonus]… you cannot do it with only your salary’. Numerous participants said that they consciously balance the work that they see as important for Mexico and/or their communities of interest with that which reaps rewards in SNI.
Some research topics that from the outside appear quite similar meet very different fates under IF-dominated incentive structures. One participant who studies social–ecological systems said that despite his strong dislike of the SNI system, he does fine within it because there happens to be a relevant journal with a sufficiently high IF score. He noted that ‘if [he] was like 15 degrees in a different research direction … most of the other interdisciplinary social sciencey journals don’t have high ratings, therefore [SNI] would become a major problem for [him].’ That there is a significantly high IF journal in his field is in part the result of fortuitous coincidence of business decisions by the corporate owners of ISI products and the citation habits and interests of authors and editors in other countries. Mexico, by building the IF into researcher evaluation, relinquishes science policy to outside actors with no connection to Mexican interests or priorities (see also Vessuri (2014)).
Several researchers noted that even on topics for which there were appropriate ISI-listed Spanish language and/or regional journals, those journals tended to be very disciplinary and focused on basic science. The collaborative and participatory projects that they felt they were hired to do typically do not get published, they noted, in standard disciplinary journals any more than they get published in ISI-recognized international journals.
Numerous scientists stated that they bucked the pressures to publish in ISI journals, or balanced those efforts with what they saw as more meaningful work. One explained as follows: ‘… we are stubborn! We are still doing both things’. That flexibility, however, is not available to everyone. Another scientist, who also saw publication expectations and high-quality research to be at cross-purposes, found that her freedom to conduct meaningful and high-quality work depended on her being in comfortable financial position:
I don’t believe in numbers; I believe in quality and it affects my salary. I don’t have children so I have the luxury to do quality research because I have the luxury to not depend on […trails off]. I don’t have a luxurious life and so money cannot control my research to the same extent as someone who has a family.
Sentiments such as these might be chalked up to typical complaints of how hard it is to be a scientist, but even scientists at the upper echelons of SNI and their home institutions offered similar examples of balancing SNI-rewarded work with ‘meaningful’ research. One SNI-3 ecologist with whom I spoke said that he had switched research areas to a field that was in fashion in high-IF journals, not because he felt it was more valuable to either the advancement of science or to potential users of the knowledge, but because he had been stuck at SNI-2 for more than a decade and the financial pressure from SNI and his university was worth making the change. There was a palpable frustration that evaluation metrics not only failed to reward, but actively devalued what the participants deemed to be important and high-quality science.
Individual institutions in Mexico typically have their own publication incentive systems and could use them to develop their own priorities and approaches to research. Many, however, do not deviate substantially from SNI definitions of ‘high quality’ research because doing so would cause them to lose funding for research and graduate fellowships, which are tied to productivity. And, were an institution to deviate strongly from SNI in terms of the research it incentivized, individual scientists within would incur the cost of foregoing SNI-recognized work to succeed in the institution’s system. Individual institutions have incentives to double-down on those SNI-based productivity policies rather than seeking to align their pay structures with their institutional missions.
Possibly in part because scientists are titrating their publication efforts to produce just enough articles in listed journals to make their needed salary bonuses, a great majority of scientists in the system are at the first level. To further boost ‘productivity’, CONACyT has created additional incentives, such as stipulating that for graduate programs to achieve top accreditation levels (and receive the associated resources), 60 per cent of the full-time research professors in that program must be SNI members and 40 per cent must be in Levels 2 and 3 (CONACyT, 2015).
Further pushing quantity of scientific production, CONACyT makes student aid contingent upon universities having students graduate within 2 years for a master’s degree and 4 years for a doctorate, and students are required to publish (preferably in a high-impact journal) to obtain their degrees. This has led some universities to formally sanction professors whose students do not graduate within that time frame. These policies, my participants explain, are systematic incentives for them and their students to avoid: field research that requires multiple field seasons; projects that could be delayed by unexpected ecological conditions or changes; and work that is not deemed to be of interest to distant editors of ISI-listed journals.
If there were a wisdom inherent in the invisible hand of science that transcended the insights and instincts of our participants, it might not constitute a problem that so many scientists felt pulled from what they deem to be ‘important’ work for their region and pushed toward that which yields frequent publication and citations. The Mexican ecological research portfolio, however, leaves some problematic gaps that indicate that the invisible hand—though a powerful steering force—does not guide researchers toward topics that benefit the sponsoring nation. Substantial scientific work has long since been conducted in the regions surrounding Mexico City because of the comparative wealth of that region and the concentration of universities and research centers there. Other regions of the country, including the most biodiverse regions of southern Mexico, historically received very little scientific attention because they are comparatively remote. Current policies create a disincentive to study these areas today: ecologists report that there are few scientists today who are willing to conduct the initial biological surveys required to establish new research sites because doing so would result in a 3- to 5-year reduction in ‘productivity’ as measured by SNI. Thus, even though research in these culturally and biologically diverse regions would probably yield high IF publications, the work is not conducted because of unintended steering influences of the SNI incentive system.
The geographic bias in research attention—which participants asserted is exacerbated by publication incentives—is evident in terms of statistical measures of scientific productivity, including SNI membership data: in 2009, the Federal District had 1,066 scientists in the SNI system in Chemistry and Biology (Area II, which includes ecology) while several southern states, with some of the highest biodiversity, had between one and fifty each. All but three of the Mexican states outside of the Federal District had fewer than ninety (San Román and Zúñiga-Bello, 2009).
Even governmental knowledge users in Mexico are frustrated by the impacts that the IF-centric researcher evaluation has had on Mexican science. Interview participants reported that CONABIO, Mexico’s biodiversity conservation ministry, is increasingly frustrated that the SNI incentivization system dissuades scientists from conducting biological and species distribution surveys and taxanomic research—especially in relatively under-studied ecosystems and regions—both because of the slow pace of that work and because the outputs they need tend to be datasets and reports, rather than the foreign language peer-reviewed journal articles that policies encourage researchers to produce. The dynamic is sufficiently strong that CONABIO has trouble convincing scientists to conduct these types of studies even when CONABIO is providing the funding for the work.
There is a similar tension about desired outputs between scientists and their funders related to a funding stream known as fondos mixtos (mixed funds) that combines federal money with support from the Mexican states. Researchers receiving this money report being caught between competing demands from the two funding sources: the states want decision tools relevant to local knowledge needs, while CONACyT and scientists’ host institutions want publications in and citations from international journals. The lack of consensus about what constitutes an important scientific output that exists between funders and scientists was sufficiently aggravating that it came up in numerous conversations with both types of actors. Some calls for proposals specifically state that a scientific publication will not count as a product of the project; funders want to see that their money contribute toward application and they do not consider a publication to be a substantial contribution in that direction. As one participant noted:
Government decision makers and the forestry commission, the national parks administration, all those people – they’re frustrated because the scientists consider the publication as their be-all and end-all, but for a policy person a publication doesn’t do anything for them. I have concrete problems in the field that you need to take care of. … The decision makers are frustrated with the scientists.
For their part, the scientists report that the incompatible expectations of those funding sources and the SNI system not only create unreasonable demands on their time, but also that the content of the work desired by the state funders is typically not of interest to high IF journals. However important it may be to Mexico, several of our participants said that they eschew fondos mixtos because the projects do not offer adequate rewards in the SNI system. It is worth noting also, however, that many scientists avoided fondos mixtos also because the specificity of the calls for proposals makes them look overtly political and possibly written with specific scientists in mind. The specificity of the calls runs counter to the scientific self-governance that these scientists are most comfortable with, even as they recognize the distorting influence of their own governance.
SNI’s impacts on scientific knowledge systems continue beyond the selection of research topics and field sites. Perhaps the most obvious impact is that SNI incentives push researchers to publish in English language journals, which presents substantial challenges to would-be users of that knowledge in Mexico. Some universities provide translators to ease the challenges for their scientists in publishing in English, but I learned of no analogous effort to translate research findings of local interest back into Spanish. Even if we were to accept the linear model’s unfounded presumption that the advancement of knowledge automatically or inevitably contributes to societal benefit, we can recognize that language barriers present a substantial burden complicating the receipt of that benefit. Some professors report that they find themselves translating key articles for their students as part of their class preparation simply because adequately high-quality research is increasingly difficult to find in Spanish language journals. Defenders of SNI justify the increasing focus on English language publication by arguing that their goal is to contribute to a global science, not an isolated regional one, and that all scientists will learn English as that becomes a necessary skill. This argument, however, assumes that the only audience for science is the established scientific community, forgetting the frequently invoked justification for science funding that it will lead to better outcomes for the funding nation. To linguistically isolate students—some of whom are striving to become scientists or ecosystem managers themselves—and decision-makers from the scientific knowledge base makes it much more difficult for society to benefit from science, regardless of the size of the research budget.
Compounding this formidable linguistic barrier is a financial one: subscriptions to individual high IF journals can cost institutions thousands of dollars or more, and individual articles for those without journal subscriptions are prohibitively expensive. Many of the scientists I spoke with outside of Mexico City reported difficulties in accessing the journals in which those researchers are encouraged to publish. To gain access, numerous ecologists reported that every time they came across a title of interest, they emailed colleagues at wealthier institutions in the capital or abroad to gain access through those library subscriptions. Adding insult to injury, many institutions also cannot afford subscriptions to WoS databases, the very subset of science to which they are expected to contribute. It goes without saying that if university researchers themselves have trouble accessing the journals in which they publish, similar barriers prevent most potential local and regional knowledge users from accessing the information those researchers produce. One scientist noted that the CONACyT budget, which he felt was already inadequate, is regularly at threat for additional cuts and attributed that vulnerability to the fact that national decision-makers saw little benefit from a scientific enterprise seemed geared toward production of English language journal articles to which few Mexicans have financial or linguistic access.
In order to publish adequate numbers of articles in ‘top-tier’ journals, many researchers with whom I spoke reported that they were forced to collaborate with one another and with scientists in wealthier countries. Doing so gains them access to the equipment, library resources, and linguistic help that high IF journals require. While these are seemingly beneficial outcomes, participants reported that the reality is more complicated. They reported that they end up trading co-authorship with one another when in fact there was little true collaboration. And, they reported deliberately committing ‘salami science’—a term they use to describe submitting ‘the least publishable unit’, splitting what should be one manuscript into several that are submitted to different journals. When disconnected from one another, any potential user of those papers has to find them all, obtain them from behind paywalls, and assemble the information into a cohesive whole. Needless to say, this approach to publishing is problematic if the goal is to produce useable science. Further, several of my participants reported surrendering their own research agendas to those of their better-funded collaborators in wealthier countries in order to participate in the types of resource-intensive research likely to be accepted into high-IF journals. Given the enormous disparities in distribution of resources available to scientists across the world, it is hugely problematic to ask scientists in resource-poor contexts to ‘compete’ in a global scientific project, especially one that is defined by journal editors unaware of the knowledge needs of scientists’ host countries.
These findings confirm what Vessuri and co-authors suggest happens throughout Latin America: scientists seeking publication in ‘core’ international journals are forced to participate in research agendas defined by distant researchers, editors, and journals, often to the exclusion of local and regional knowledge needs and interests (Vessuri et al., 2014). In the case of Mexico, the desire to publish in those core journals derives not just from scientists’ desire for prestige, but it is codified in policies that frustrate many of the scientists with whom I spoke.
4. Concluding thoughts: science and solving problems
Science policies are products of political processes rooted in cultural notions of ‘good’ science. In the USA, many of the main science policy documents of the postwar period have extolled a model of scientific self-governance as yielding the most efficient advancement of science, and essentially assumed that scientific productivity in the form of publications and citations automatically yields societal benefit. This thinking now permeates science in Latin America as well (Vessuri et al., 2014). There are numerous problems with this model, including the well-documented one that it does not encourage effective uptake and use of the resulting knowledge by potential users. Equally problematic, though, is that science policies that focus on increasing productivity neglect the fact that not all science is created equal. In ecological research, where there are innumerable scientific questions that could be asked of a huge range of ecosystems and species, not all research agendas, approaches, or formats serve societal (or scientific) goals equally well. Even the policies designed only to boost productivity with no intent to steer science end up incentivizing some types of and approaches to research to the detriment of others. Vessuri et al. (2014) argue that when scientists seek to publish in top international journals, an outcome codified in Mexican science policy, the journals themselves:
Collectively, through their editorial policies, decide what questions are important, and thus create a kind of collective, flexible, largely unplanned, yet narrowly controlled, form of science policy for the world. It is this collective device that, incidentally, largely accounts for curious knowledge gaps such as neglected diseases. (Vessuri et al., 2014)
In the case of Mexico, science policies under SNI encourage exactly that: they hand science policy decision-making authority to distant editorial boards, and in the process direct research attention away from known national knowledge needs.
It is understandable that sponsors of science in Mexico and elsewhere want to ensure that they are benefiting from the resources they pour into that work, but that which is most easily quantifiable about science—publications, citations, and patents—is not inherently related to the societal benefit that the society expects to receive (Bozeman, 2003; Bozeman and Sarewitz, 2011; Meyer, 2011). Building science policies around these quantifiable indicators directly attaches Mexican research attention to the selective pressures associated with international publication dynamics. This builds a strong rudder into the systems steering science, but that rudder actively dissuades ecologists from addressing known knowledge needs.
This article documents that Mexico’s SNI discourages ecological research that is risky, slower, or ‘merely’ of national or regional interest, and it pushes scientists to publish in a foreign language in inaccessible journals. And although many of the Mexican scientists we spoke with desire to establish the ongoing two-way dialogue with knowledge users that current scholarship suggests will yield more effective use and uptake of scientific information, SNI systematically penalizes that approach to science. Even the scientists with whom we spoke who have a more linear understanding of the relationship between knowledge creation and utilization have experienced unintended steering effects of SNI and report that it undermines the quality of Mexican ecology.
The SNI system had its genesis in crisis, and it served an important purpose at a critical point in Mexican history. Its current implementation, though, is problematic. It does not have to be this way: Latin America is leading global efforts to create alternative scientific databases and encourage open access publishing (Vessuri et al., 2014). Latindex, Redalyc, and SciELO are databases designed to provide more comprehensive coverage of the Latin Amercan region. As noted in the Section 2.3, Latindex lists 5,408 active periodicals in Latin America ‘of academic interest’, as compared to 242 included in the 2010 SCI (Alperin, 2014). Publication reward systems built upon publication in Latindex journals would thus likely constitute a substantial improvement to the problems documented in this article and would better represent the work currently being done. WoS-based evaluation is really a metric of the extent to which researchers are contributing to the research interests of the world’s wealthy countries, as filtered by profit-oriented editorial decisions by corporate owners of that database (c.f. Vessuri et al., 2014).
Within Mexico, reorienting publication incentives could happen at a number of scales: SNI area committees could rework their point allocation structures to reward publication in regional journals, and universities and research institutions could do the same. Were regional journals, indices and associated open access publication efforts to be accepted, the uptake and use of scientists’ products would not be hindered by financial barriers. And to the extent that these journals continue to accept Spanish language submissions, linguistic barriers to knowledge uptake and use could be minimized. And importantly, Mexico could retain control of its own research agenda and science policy. The talented scientists of Mexico could be contributing articles to educate the next generation of scientists in the region and decision-makers would have better access to their work.
Recognizing that a citation in a scientific article is only one of many potential indicators that a particular article has had an ‘impact’, a number of prominent efforts have emerged—and gained vocal support in Latin America—to develop alternative metrics of research importance (altmetrics; see Alperin et al., 2014). As authors of a recent report on these efforts note, however, a major limitation remains: the main altmetrics being developed still treat articles as the unit of evaluation and thus exclude a substantial portion of research in Latin America, which is communicated in forms other than peer-reviewed publications (Alperin, 2014). A number of my interview participants noted that government ministries and decision-makers frequently have no use for peer-reviewed publications; what they need most from any given project may be reports, evaluations, and community presentations. If these are the products most desired by potential users of the research, they should be captured in evaluations of researcher contributions.
Researcher evaluations constitute one of the more influential forms of science policy in that they have systematic steering influences, shaping both the subjects and methods of research. The growing efforts to develop alternative metrics of scientific impact or contribution (altmetrics) are improvement over citation-based metrics, especially those that are built around databases and indices with strong biases against relevant languages. I would like to push further: to ensure flexibility that would allow scientists to explore the more biodiverse regions of the country, to work collaboratively with the communities who stand to benefit from scientific knowledge, and to produce the knowledge that government decision-makers actually need, policies must not strictly be oriented around the quantification of publication outputs.
A number of authors have argued that in our complex world, successfully addressing problems that simultaneously contain technical and value-based components—such as management of natural resources—will require a new approach to research conduct and evaluation (Funtowicz and Ravetz, 1993; Nowotny, 2003; Nowotny et al., 2006; SPARC, 2010). Robertson and Hull (2003), for example, advocate for an approach to ecology that directly engages with stakeholders in a collaborative fashion in the service, making the most of democratic and scientific processes. This engaged approach of collaborating in long-term iterative relationships with potential knowledge users is well accepted by many of the Mexican ecologists with whom I spoke. It is not, however, compatible with the publication expectations within which they work: time spent working with communities on the questions of interest to them is time not dedicated to advancing research on topics of interest to editors of journals, especially prestigious international ones. Publication expectations rooted in an outmoded understanding of the science–society relationship serve as powerful structural barriers to attempts to better link scientific advancement with societal benefit. Thoughtful researcher evaluation itself requires time and nuance. As Philip Altbach has said, ‘It is probably too much to ask that care, discretion and sophistication be used when making judgements that often affect the salaries and academic futures of professors in an age of hyperaccountability’ (Altbach, 2014), but that is exactly what is needed if nations are to benefit from the research that they sponsor.
The tendency within science to reduce the impact of science to that which is easily quantifiable and optimized—namely, publication and citation in internationally recognized journals—is strong (Vessuri et al., 2014). Tellingly, despite the presence of innovative regional databases and indices and the strong dissatisfaction with ISI products, few of my participants were familiar with these alternative databases when asked. This is likely in part because they are not prominently featured in the SNI and institution-specific policies that directly affect their pay, and it speaks to the extent to which the ISI IFs have come to define high-quality scientific work for scientists worldwide. At the same time, however, my interviews also reveal that Mexican ecologists have not yet fully accepted the intensive quantification of productivity as inherent to high-quality science. Since the area committees within SNI have authority to alter their criteria for researcher evaluation those scientists retain substantial power to improve the situation. Scientists themselves, as arbiters of prestige within science, are accurately considered to be science policy makers (Neff, 2011; Miller and Neff, 2013). There is currently a window of opportunity for scientists and other science policy makers to reconsider how scientists are evaluated; that window may close as scientists receiving their training today are acculturated into a production-oriented scientific culture.
Acknowledgements
This research would not have been possible without the generosity of the thirty people who gave an hour or more of their time to answer my questions. I would also like to thank Sharon Zavala, Ziania Narvaez-Garcia, Samara Almonte, and Alyssa Rooks for their work as research assistants. Thanks also to three reviewers and the journal editors for their thoughtful and extensive comments on a prior draft. This work was supported by the United States National Science Foundation (www.nsf.gov) [Grant Nos. 1465279 and 1158723]. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the author and do not necessarily reflect the views of the National Science Foundation.