What if your paper were retracted for no credible reason?

Abstract

Retractions are meant to protect the integrity of the published record against erroneous content, but retraction procedures are not infallible. This paper identifies imperfections in the regulatory framework of retractions, draws conceptual lessons from them, and derives recommendations to improve the handling of disputed retractions to better suit the needs of all stakeholders. The imperfections are illustrated by a retraction, which its authors and prominent members of the research community appealed at various levels to no avail. None of the relevant authorities in the system of retractions acknowledged any problems, although the retraction suffers from a number of flaws, both procedural and on merit. This paper concludes with reflections on how to revamp the framework for retractions in scientific journals.

1. Introduction

Scientific publishing is based on trust: the trust of readers that published articles are thoroughly peer-reviewed, the trust of reviewers that their pro bono work makes a difference, the trust of authors that editors act in good faith, and the trust of research funders that the publication system is reliable. In particular, authors trust that their published article will not be retracted just because it stepped on someone’s toes. Authors trust in the accountability of those who make publication and retraction decisions, and that if a mistake is occasionally made, the respective authority will correct it. However, in complex social systems, trust can be difficult to build but easy to break down, if the foundations on which trust rests become shaken or even destroyed.

Retractions are essential for maintaining the integrity of the published record: ‘Retraction is a mechanism for correcting the literature and alerting readers to articles that contain such seriously flawed or erroneous content or data that their findings and conclusions cannot be relied upon’ (COPE Council 2019a: 3). In fact, retraction is the most punishing instrument in the editor’s toolbox. Rejections of submitted manuscripts are undisclosed, happen all the time to everyone, and do not ruin anyone. But retractions are different in that they are public, relatively rare, and attach a stigma to the author for life. Once published, a paper becomes part of the scientific literature and should be protected unless it is proven beyond reasonable doubt that something is seriously wrong with it. Editors cannot afford to retract a published paper recklessly.

With this in mind, it is alarming that, as showcased in this paper, some actors in science publishing believe that they are at liberty to use retractions creatively to advance their own interests, and mechanisms that are supposed to be in place to hold them back do little, if anything at all, to remedy the situation. Business and other third-party interests have long exploited opportunities to supress inconvenient research. What is different today, however, is that these challenges come from within the scientific publication system and there are worrying signs that such interests have begun to exploit retractions to that end. If this becomes more common, scientific discourse and academic freedom at large are in danger. Scientists could lose trust in the publishing system and hesitate to write critically. In fact, this could be already happening.

Although research on retractions is expanding (for recent surveys see, for instance, Hesselmann et al. 2017; Kohl and Faggion 2023), existing studies are limited to what can be relatively easily observed in bibliometric databases and retraction notices, such as the reasons cited for retractions, who requested them, characteristics of the journals, origins of the authors, and the consequences of retractions, which are typically in terms of citation rates. Nevertheless, there has been little research on the process of decisions to retract, except in terms of the time to retract, and to my knowledge, there is no study that has examined whether retractions are actually justified, or what happens if they are not. This may be because this requires deeper understanding of the context, which is difficult to obtain.

Hesselmann et al. (2017) and Xu and Hu (2021) report that most papers are retracted by the authors, which would signal that most retractions are just. The Retraction Watch (2023a) archive confirms that retractions usually happen upon the request of—or at least in agreement with—the authors. Retractions against the will of all or some of the authors are less common and, in most cases, pertain to disputes over whether a mere correction would suffice to rectify a mutually recognized error and/or revolve around issues with plagiarism, access to data, and the authorship itself. Retractions that the authors claim are completely unjustified due to a lack of evidence that the findings in the paper are unreliable and that the authors deem to be driven by business and other third-party interests are evidently rare. However, such cases should not be overlooked, because consideration of them may uncover flaws in the retraction system that will otherwise remain hidden but that need to be fixed precisely to prevent them from becoming more prevalent.

The purpose of this paper is to help improve the regulatory framework of retractions by making four main points. First, the Committee on Publication Ethics (COPE), which is usually referred to as the system-level authority to settle disputes over retractions, devotes insufficient resources to examining complaints, and therefore risks being unable to fully grasp a complex case and reach a fair resolution. Second, the procedures of some publishers, including one of the largest ones, to resolve disputed retractions may not be fully developed to meet the needs of all stakeholders. Third, conflicts of interest, which can arise either in relations between publishers, or with COPE as regards to publishers, can affect retractions and the complaint process. Finally, ethical standards apply to editors, journals, and publishers equally as they do to authors, but authors tend to endure far stricter scrutiny than the others. As a result, authors of articles that have been wrongfully retracted can find it difficult to be heard.

This unfortunate dynamic is illustrated by the remarkable retraction of Macháček and Srholec (2021) in Scientometrics, a highly respected field journal published by Springer Nature. This paper discusses the breach of the established retraction guidelines in this case and, most importantly, the futile strive for a remedy (see also Retraction Watch 2021, 2022; Srholec 2021; Abramo et al. 2022). The authors and their supporters appealed to the editor, publisher and COPE, neither of which acknowledged any problem, not even a factual error in the retraction notice, and did not produce a reasonable resolution. This retraction provides an unwanted—though in hindsight, valuable—test of the complaint mechanisms, and the fact that the relevant authorities failed to see the missteps exposes a serious problem in addressing disputed retractions. The paper draws conceptual lessons from this evidence and derives recommendations at various levels on how the regulatory design for handling retractions should be revamped to better suit the needs of authors and the research community as a whole.

2. Taking stock of the regulatory design

COPE has been designated by its member publishers, journals, and editors—and is generally understood by the research community—as the authority to prescribe rules of retractions and oversee their implementation at the system level. Major publishers, including Elsevier (2023), Springer Nature (2023a), Taylor & Francis (2023), Wiley (2023), Sage (2023), and Emerald (2023) refer to COPE for retraction guidelines, and in some instances for complaints and appeals. Likewise, numerous journals, including Science (2023), Nature (2023), Lancet (2023), and PNAS (2023) refer to COPE for guidance on editorial practices.

2.1 Description of the COPE guidelines and complaints process

According to its own webpage, COPE’s purpose is to ‘educate and advance knowledge in methods of safeguarding the integrity of the scholarly record’, and one of its missions is ‘offering a neutral, professional voice in current debates’ (COPE 2023a). The Facilitation and Integrity Subcommittee of COPE has been established to facilitate the resolution of disputes and to consider complaints against its members in a fair and objective manner. However, there are disclaimers that it ‘does not undertake investigations’, ‘will review procedural matters’ and ‘cannot make determinations on individual editorial decisions’ (COPE 2023b).

COPE has issued retraction guidelines (COPE Council 2019a) that stipulate that editors should consider retracting a publication when ‘they have clear evidence that the findings are unreliable, either as a result of major error (e.g. miscalculation or experimental error), or as a result of fabrication (e.g. of data) or falsification (e.g. image manipulation)’ (COPE Council 2019a: 2); other relevant reasons for retractions include plagiarism, copyright infringement, reporting unethical research, compromised or manipulated peer review, or failure to disclose a major conflict of interest. Retractions are not appropriate if ‘the main findings of the work are still reliable’ or ‘an editor has inconclusive evidence to support retraction’ (COPE Council 2019a: 3).

The guidelines also prescribe procedural matters. The reasons for a retraction are explained in a retraction notice, which ‘should be limited to proven facts’ and ‘should not engage in speculation’ (COPE Council 2019a: 7). Editors ‘should negotiate with authors and attempt to agree on a form of wording that is clear and informative to readers and acceptable to all parties’ (COPE Council 2019a: 4). If the main findings hold and “if only a small part of an article reports flawed data or content, this may be best rectified by a correction” and “if conclusive evidence about the reliability of a publication cannot be obtained… retraction may not be appropriate, but an editor could consider publishing an expression of concern (COPE Council 2019a: 3, 6). And ‘since responsibility for the journal’s content rests with the editor, they should always have the final decision about retracting material’ (COPE Council 2019a: 5). In other words, the threshold for retraction is rather high, and retraction should come into play only as a last resort.

Differences in views about scientific approaches, methods, or data are a natural element of science that should never constitute a reason for considering a retraction. In fact, disagreements are the motor of science and need to be preserved as part of the scientific record. What has become obsolete is not retracted, but is superseded by more accurate results. Only misconduct or wrongful application of methods justifies retraction, and there must be no reasonable doubt about the flaws that result in a retraction. If there is not a clear consensus, retraction should not be an option, as otherwise scientific discourse is in danger.

Anyone who deems that these principles have been violated by a member of COPE, which includes the vast majority of respectable scientific journals, is entitled to submit their concern to the Facilitation and Integrity Subcommittee of COPE. Among other things, the complainant is asked to summarize the issue, explain what aspects of the guidelines have been contravened, and describe what outcome is sought from this process. The complainant must agree to refrain from commenting on social media or blogging sites while the issue is active. The committee then reviews the concern. If the committee finds the concern relevant, the committee asks the member to respond to points posed about the concern, and concludes the review by providing a written summary with advice and guidance on how to resolve the issue to the complainant and the member.

According to the COPE sanctions process (COPE Council 2019b), if the review concludes that major concern about a member’s practice is justified, but the member does not respond adequately to recommendations on how to resolve the issue and ‘demonstrates a flagrant or consistent unwillingness to abide by COPE principles’ (COPE Council 2019b: 1), the incriminated member would be sanctioned by having its membership placed on probation. If continued remediation does not resolve the problem, its membership would be revoked. Hence, there is an enforcement mechanism in place that, if escalated all the way to public loss of COPE membership, could deliver reputational damage.

2.2 Contradictions in the COPE dispute settlement mechanism

COPE (2023b) lowers the expectations of a potential complainant by emphasizing that it is not a regulatory or statutory body, and that its primary role is ‘not to adjudicate complaints’; although anyone is welcome to submit a complaint and a COPE review is expected to produce advice and guidance on the issue, presumably by ruling whether its guidelines were contravened or not. Such a ruling should resolve the dispute, which is why the complainant submits it in the first place, so to explicitly deny the role of adjudication is confusing. Taken together with the three disclaimers—‘don’t investigate’, ‘review-only process’ and ‘don’t look at individual decisions’—one is tempted to ask what the design of this complaint process is suited for?

First, COPE needs to get to the bottom of any given case to be in a position to assess whether something in fact went wrong. COPE is not going to be able to assess that if it does not undertake an investigation. If there is a factual discrepancy between the statements of the complainant, typically an author, and those of the defendant, typically an editor, which should be expected in disputes, the review needs to determine which party is correct. Without an investigation, a review is bound to side with one or the other party without a concrete reason.

More precisely, without an investigation, a review is able to resolve only banal cases, in which one side makes a simple mistake, usually because it does not know the guidelines, and when the incriminated side is ready to acknowledge the error. But review without investigation is toothless for uncovering more complex cases of editorial misconduct, in which an author seeks justice against a senior editor who knows the rules and decides to cover up wrongdoing. Hence, this review process is likely to serve editors, who can rely on the notion that reviewers will not dig deeper, to check whether their response to review queries is truthful. However, authors who file a complaint are likely to be poorly served by this process.

Second, COPE makes it clear that their review will assess procedural matters, thus focusing only on whether prescribed steps have been taken in the correct order, but not the substance of these steps. At the heart of its retraction guidelines, however, there is the obvious requirement that a retraction must be justified by merit. By not assessing whether there is clear evidence that the findings are unreliable, either as a result of major error, fabrication and falsification, or whether there are other significant reasons for retraction, such as plagiarism, the review fails to resolve disputes over retraction that refer to this core principle. Without investigation into its substance, the review will praise a retraction as flawless, if the prescribed steps were nominally taken, even if the editorial decision could be baseless.

Thus, what seems to matter for the review is whether the right kind of statement was issued at a given point and eligible parties were notified of it, and whether the correct rubber stamp was used to seal the deal. But the review will not examine, for example, whether the editor manipulated post-publication assessment of the manuscript to create a pretext for retraction, or whether the reasons cited in the retraction notice are speculative or clearly nonsensical, even if the authors submit a complaint describing exactly such a problem. Reducing the review process to fulfilling formalities is, if not dangerous, far less than what anyone would expect from a mechanism that is geared to assess violations of retraction guidelines.

Third, there is the disclaimer that COPE does not make determinations on individual editorial decisions. What else does COPE aim to make determinations about, if an author complains about a retraction, which results from an editorial decision to publish a retraction notice tagged to that manuscript? Or does COPE not review complaints about individual retractions at all, in which case its complaints process is unsuitable for authors, who typically seek answers about their particular manuscript, rather than information on general procedural matters of journals, which in fact are often hidden to authors? Could it be that COPE is equipped to deliver far less with regards to dispute resolution than what its member publishers, journals, and editors want authors to believe?

2.3 Conflicts of interests of COPE and the publishers

According to Wilmshurst (2022), five major publishing houses provide about half of COPE’s income, with Springer Nature being one of its largest (if not the largest) funders. As a result, COPE has a major conflict of interest with regards to publishers. So long as COPE is dependent on major publishers for funding, it is wishful thinking that it could objectively assess complaints about journals with which they are associated. How likely it is that COPE would side with authors in a complaint against a journal, if down the line, this could result in a reduction of its budget? In this context, how credible is COPE’s commitment to sanction a member by revoking its membership, if it demonstrates unwillingness to abide by COPE’s principles of publication ethics (COPE Council 2019b)?

Moreover, conflicts of interest do not necessarily end there. Publishers are private entities that do not exist in a vacuum. Some, particularly major ones, are complex business constructions that are ultimately owned by investors. These investors not only own the publishers, but can also have interests that compete with evidence presented in journals across science fields and topics. Some publishers are cross-owned directly with each other or indirectly through investors, which further creates an intricate web of business relationships that might not be apparent to outsiders, but that can trickle down to assessment of scientific work at various levels. Unfortunately, this could transpire, for example, when a retraction is instigated by a publisher that has aligned business interests with a publisher of the journal that published the manuscript.

In concert with at least some publishers, as documented below, COPE downplays, this type of conflict of interest; perhaps because they are well aware of how damaging this could be to the credibility of their decisions. Nevertheless, this is in sharp contrast to how seriously the issue of competing interest is taken when it comes to authors. In fact, an author’s failure to declare a conflict of interest, as already noted above, is one of the few justifications for retracting a manuscript. But no sanction is defined anywhere for a journal, publisher, or COPE itself for failing to declare a conflict of interest when making decisions regarding the work of authors.

Senior editors and other professionals that are involved in making decisions in its member journals and publishers are supposed to be knowledgeable about the COPE rules. By corollary, when they are confronted with a disputed retraction, they could also get to know—or inform each other, if need be—about the contradictions in the COPE complaint process and the aligned interests between COPE and publishers. This leads to a moral hazard situation, in which they are more inclined to take risks such as retracting a paper unjustly, because COPE is likely to cover their backs if the authors complain. A COPE ruling that absolves them goes a long way towards protecting their credibility in the research community. In this respect, COPE is part of the problem, which makes it even more important to debunk the myth that COPE is in a position to cater equally to its members and to outside stakeholders, including authors.

2.4 Limits to self-regulation of editors and the community

Editors are assumed to be independent, impartial, and to protect the journal from third-party interests. But that does not rule out that rogue elements can exert pressure on them to bend the rules. Editors can find themselves in an unenviable position, in which they have to navigate between the interests of authors and those of the publisher, which can be a tricky test of their editorial integrity. On the one hand, editors have the final authority to decide about retractions. On the other hand, however, they are in one way or another contracted to the publisher, and in practice they need to play ball, because their editorial chair could be at stake. Not every editor is ready to enter into a conflict with a publisher over a single paper being considered for retraction, especially in high stakes situations. Relying overly much on the ability of editors to withstand these pressures is risky from the systemic perspective.

The last line of defence against unethical publishing practices is the research community around the journal, which can rally behind dishonoured authors. However, this type of action also cannot be taken for granted. First, there may be some degree of apathy. Both higher education and academic research have experienced massification, weakening the sense of belonging and joint responsibility. Second, scholars are often reluctant to move against journals and their editors, especially those that have accepted their manuscripts for publication in the past. This is particularly likely for field journals in narrow or newly emerging areas, and is exacerbated by performance-based evaluation systems featuring a ‘publish or perish’ culture.

Finally, the risk calculations of editors and how the research community reacts depend on who the potential victims are. For example, authors who are not well networked in a particular field, such as junior scholars, scholars from related fields, or scholars outside of core universities and geographic areas are far more vulnerable than their more established and better positioned counterparts. Needless to say, however, the guiding principles ought to apply equally to everyone regardless of status or origin. The main guarantee of that is expected to be a stratum of formal, independent, and accessible checks within the scientific publication system. This paper questions the efficacy of such checks.

3. The case in point

To illustrate these dangers, I present the case of a paper on predatory publishing written jointly by myself and Vít Macháček that was retracted recently. We began to write the retracted paper in early 2016. The first output of this work came out as a policy paper (Macháček and Srholec 2017). Because the results attracted the attention of academics and stakeholders, we decided to develop the analysis into a journal paper. The updated version of the paper was submitted to Scientometrics, one of the leading carriers of science communication in that field of study, in June 2019. The paper went through a rigorous peer review with two rounds of major revisions and was published in that journal under the title ‘Predatory publishing in Scopus: evidence on cross-country differences’ in February 2021 (Macháček and Srholec 2021).

3.1 The unjustified retraction

In May 2021, the Chief Executive Editor of a major open access publisher complained about the article to Scientometrics, requesting that ‘this article must be immediately retracted’, essentially because the name of the publisher was used in this research in the context of predatory publishing, which was not good for their business (see the full text of the retraction request for download from Retraction Watch 2021).

The Editor-in-Chief of Scientometrics, did not fend off this attack, but instead organized a post-publication peer review, from which dubious claims about flaws in the paper were derived. After ignoring counterarguments from the authors and amid calls from prominent members of the research community for more scrutiny, the editor retracted the paper without further deliberation. The retraction notice was published online in early September 2021 (Macháček and Srholec 2022a).

The retraction was seriously mishandled in terms of merit and procedural matters on multiple accounts:

1. Initially, the editor decided that the case would be best resolved by publishing the complainant’s concerns as a letter to the editor. After the complainant refused this solution and insisted on the retraction, the case should have been closed. Instead, however, the editor began to investigate the paper.

2. The editor did not respond to detailed counterarguments of the authors at any point; there was no meaningful discussion about the merit of the retraction with him. The editor gave one week to the authors to respond to the retraction request stating that ‘the basic concern regards the use of “Beall’s List” as data source.’ The authors sent a detailed response in due time that addressed the complainant’s arguments, in which they argued that Beall’s lists were the dominant source of data in this line of research (at that time) that was used in a number of other papers recently published in highly respected journals and concluded that none of the COPE’s conditions for a retraction have been met. There was no reply by the editor except of his acknowledgement of receipt (see the full text of the authors’ response to the retraction request in online Supplementary File S1).

3. The post-publication peer review was unbalanced, not representative of the literature on this topic, and thus unsuited to provide fair assessment of the manuscript. The editor gave two weeks to the authors to respond to his summary of the reviews. The authors sent a detailed response in due time, which provided detailed answers to the comments point by point, but also pointed out that ‘The post-publication peer-reviewers seem to be strongly critical about the use of Beall’s lists… It would be fair to consult reviewers who are ex-ante neutral, who do not have predetermined opinions in this regard, who in particular did not publish papers critical of Beall’s lists by themselves in the past, and who could therefore provide assessment… that is representative and free of emotions’. Again, the editor did not provide any reply except of his acknowledgement of receipt (see the full text of the authors’ response to the post-publication reviews in online Supplementary File S2).

4. The editor did not present to the authors conclusive evidence that the main findings of the paper are unreliable due to a major error, fabrication, or falsification and did not put forward any other legitimate reason to justify the retraction, but decided to retract the article anyway.

5. The retraction notice is speculative, uninformative, and misleading. The first reason cited to justify the retraction with regards to lacking a control group refers to a standard not established in this area of research, and the second one with regards to languages is itself a factual error.

6. Even if the alleged flaws, from which the editor claims the paper suffers, were true, which they are not, as explained in more detail below, the main findings still hold, so a correction would be sufficient to rectify the problem.

7. The editor failed to disclose a major conflict of interest due to a joint ultimate owner of the complainant and Springer Nature, which implies that they share aligned business interests.

Regardless of what one thinks about the literature on predatory journals, and the difficulty of identifying them, the way this retraction was administered represents a dangerous precedent of tinkering with the published record to appease third-party interests. Each one of the seven missteps above should alone be sufficient grounds to question the retraction. Together, they paint a bleak picture of (at best) editorial arrogance and (at worst) possible editorial misconduct, which should spur to action everyone in a position of responsibility in the journal, publisher, and the research community at large.

It is not only the authors who believe this. 27 members of the Distinguished Reviewers Board of Scientometrics and/or recipients of its Derek de Solla Price Medal, including some of the most cited researchers in this field, strongly criticized the retraction, concluding that the retraction notice is baseless, that the process applied to reach the decision to retract lacks transparency, and that key questions surrounding the retraction are still waiting to be answered; they also warned the editor that ‘closing the debate by retracting the paper is highly problematic and poses a serious threat to academic freedom’ (Abramo et al. 2022: 2).¹ Yet the editorial response by a the second Editor-in-Chief, who in the meantime became a co-editor of the journal, did not acknowledge any wrongdoing whatsoever (Zhang 2022).

Explaining each of the editorial missteps in more detail is beyond the scope of this paper (for an extended elaboration on the retraction procedure see Srholec 2021). However, the retraction notice deserves closer attention, because this is where one expects to find the main arguments of the editor to justify the retraction. What he did not state there is arguably not significant enough to mention. The retraction notice is cited by convention as Macháček and Srholec (2022a), even though the authors of the paper did not write any of it. The authors in fact strongly disagree with its content and consider it confused. It claims that some of the findings in the paper are unreliable for two reasons:

1. ‘Findings are based on a regression analysis; however, this analysis did not include a control group… In this context, the Scopus database cannot be considered a control group, since it is a comprehensive bibliographic database.’

2. ‘Similarly, the analysis was restricted to publications in four languages (English, Spanish, French, and Arabic). The results of the regression analysis are inconclusive for publications in languages not included in the analysis.’

First, the argument about the lack of a control group might sound compelling to researchers in the areas of natural science, in which controlled experiments prevail, but not in social sciences (specifically scientometrics), in which such experiments are relatively rarely possible to conduct, especially at a national level. In the cross-country regression framework used in the paper, the inclusion of a control group, however defined², is not a necessary condition for reaching sound results. In this respect, the authors followed econometric approaches that are standard practice in their line of research. Hence, the lack of a control group is not a legitimate reason to retract the paper.³

Nevertheless, one cannot exclude the possibility that the editor came to the conclusion that the time has come to establish a standard that requires scientometrics analyses to use control groups in experimental designs, as otherwise their results are too unreliable. When re-examining the retracted paper, perhaps the penny dropped and it dawned upon him that this is necessary. The retraction was then aimed to deliver the message to this research community that, from now on, the journal is going to uphold this new, higher methodological standard. Unfortunate for the paper, but possibly a great leap for scientometrics.

But if this were the case, the requirement to use a control group should apply to all papers considered for publication in the journal at least from the moment when the retraction notice first appeared online in early September 2021. Consequently, papers published since that time that do not meet this standard should also be retracted, and the journal should not publish any more recent paper that suffers from the same flaw. However, this has not been the case. This creates the appearance of a double standard: one that was highly demanding for the retracted paper, and another that is far less pedantic for other works.

Moreover, and perhaps most importantly, the main findings of the retracted paper do not actually rely on results of the regression analysis, but were already derived from descriptive evidence on the cross-country patterns. The regression analysis was added to the paper by request of one of the reviewers, with the aim to check the robustness of the main findings of interest by controlling for other relevant factors in a multivariate regression framework. The main findings hold, and, for instance, the abstract of the paper remains the same, even if the regression analysis were entirely absent from the paper. Note that running regressions is in no way a prerequisite for publishing in the journal, as most papers that continue to come out in it present descriptive evidence with the help of tabulations and figures, without the use of econometrics.

Second, the argument about languages is factually invalid. The analysis presented in the paper does not consider the language of publications, as alleged by the editor, because it is based on all data available in the Scopus citation database, irrespective of language. Instead, the cross-country analysis takes into account languages frequently spoken in the countries where the authors of papers in the dataset are based, given their affiliations. In this regard, four languages—English, Spanish, French, and Arabic, among which English is essential⁴ – are considered, because other languages are not spoken in a sufficient number and variety of countries. For example, considering German, Chinese, or Russian, which are frequently spoken only in the country of origin and at best a handful of nearby countries, would be very similar to looking at individual countries and/or highly correlated to geography, which is examined elsewhere in the paper. Hence, this is simply a false claim of the editor who misunderstood the paper.⁵

If the first reason cited for retraction relies on an imaginary methodological standard that in reality has never existed in this area of research, and the second reason with regards to the languages is factually incorrect, both reasons presented in the retraction notice to justify the retraction are unsound. It follows that, in accordance with the general principle of maintaining the integrity of the published record, the retraction notice should itself be withdrawn (or retracted) and the paper reverted to its previous published status. However, the journal does not signal any intention of doing this.⁶

3.2 Lessons from the COPE appeal

Because Scientometrics does not have an appeals procedure and Springer Nature has a conflict of interest with the complainant, we were advised by colleagues who criticized the retraction to submit our case directly to the Facilitation and Integrity Subcommittee of COPE for an independent review. I followed through on behalf of both authors in mid-October 2021. In the submission, we asked COPE to assess whether Scientometrics contravened its retraction guidelines on the grounds itemized above, each of which we explained in detail and backed up with an attachment with supporting information. Most of this submission refers to the process by which the retraction decision was reached, which is within the bounds of what even the limited ‘procedural matters only’ resolution mechanism is supposed to handle (see the full text of the COPE submission in online Supplementary File S3).

In mid-January 2022, COPE delivered a brief ruling with the conclusion that ‘the journal followed an adequate process to handle the concerns raised about the publication’ (COPE 2022), and in their accompanying email pronounced that the process was consistent with COPE expectations to reach a decision on steps required to correct the published record.⁷ In a nutshell, the COPE review did not find anything problematic about this retraction whatsoever, as though every part of the decision was made completely according to its retraction guidelines and the best practices that it educates its members to follow. COPE gave its official stamp of approval to retracting a paper in this manner.

During the review, COPE asked the journal easy-to-answer questions, and enquired into only three of the seven counts cited to support the complaint. In particular, COPE did not look into whether the post-publication peer review was biased or whether the retraction was justified, even though its own retraction guidelines stipulate that a retraction must be merit-based and backed by clear evidence. COPE took the answers provided by the journal at face value and closed the review, absolving the editor from any wrongdoing. COPE thus put full trust in the information provided by the journal and believed the editor, while it disregarded competing information provided by the authors. For all practical purposes, COPE did not properly scrutinize the complaint.

In retrospect, one could view this submission as a test—a ‘sting operation’—of the COPE dispute resolution mechanism, even though it was obviously not intended to play out that way. The retraction suffers from so many problems that must be apparent to anybody knowledgeable who cares to look into it that not showing concern regarding what has happened, not even with the plain factual error in the retraction notice, is arguably no different from a science journal accepting a fake paper for publication. Needless to say, COPE did not pass this test with honours.

It is disconcerting to realize that, an unscrupulous editor can retract a paper without proper justification with impunity, unless he or she is stopped from within a journal, which cannot be taken for granted. If the author appeals to COPE hoping for a fair resolution of the dispute, no investigation of what has really happened takes place. The editor can safely assume that a carefully worded, if not evasive, answer to the COPE review queries, that makes it look like the procedure was handled according to the guidelines, regardless of substance, would suffice to dodge the bullet, because nobody checks the validity of the arguments.

Judging from this case, COPE is struggling to fulfil its role of providing independent guidance when it comes to disputed retractions. Complaining to COPE proved to be effectively the same as complaining to the editor who is responsible for the misdemeanour in this case. In hindsight, it is clear that the complaint to COPE was not a productive way forward, because waiting for its results only delayed pursuit of other avenues to tackle the problem, and raised false hopes of the authors for a fair resolution. But who should the authors turn to, then?

3.3 Lessons on the journal publisher springer nature

Much of what has been said about COPE in terms of its failing to hold journal editors accountable for deviating from the established code of conduct also holds for Springer Nature, which should provide the first line of defence. The Springer Nature Research Integrity Group (SNRIG) ‘advises editors, reviewers, and authors on best practices and ethical conduct in research’ and ensures everybody that ‘we take a positive and proactive approach to preventing and addressing errors and misconduct’ and ‘in complex cases we will also undertake any investigation required to resolve the case’ (Springer Nature 2023a).

Springer Nature, including its senior officials and SNRIG, was fully informed about the retraction in our case and had plentiful opportunities to address the missteps. The editor made no secret that SNRIG guided him throughout the retraction process, so it is clear that the publisher was involved from the outset. Academics supportive of the authors got in touch with one the editorial directors at Springer Nature about the upcoming retraction. Further, SNRIG’s Research Integrity Manager was included as a representative of the publisher in communications during the COPE review. Yet the authors are not aware of any steps taken by the publisher to correct the unsubstantiated claims in the retraction notice or to address the procedural misdemeanours of the editor itemized above.

In addition, in the second half of November 2021, the Czech Academy of Sciences, the employer of both authors, in a high-level communication to Springer Nature, expressed strong disagreement with the retraction and serious doubts about its justification, and called for the retraction notice to be withdrawn. In the first half of January 2022, before the COPE ruling had arrived, the Chief Journals Officer of Springer Nature replied that they ‘are not aware of any evidence that appropriate due process was not followed’ and recommended that we submit the case to the Facilitation and Integrity Subcommittee of COPE, which was already closing its review at that point, because ‘this should give you confidence that your concerns are being assessed by an appropriate, expert and independent body’ (Srholec 2021).

What else and from whom does Springer Nature need to hear to start a serious internal enquiry about possible editorial misconduct? How this case unfolded indicates that even a leading science publisher may not have a well-functioning mechanism to investigate disputed retractions, which surely would have uncovered at least the factual error in the retraction notice. Instead, the publisher forwarded the complainant to COPE, which does not undertake investigations. Just like COPE, SNRIG backed the editor and disregarded the authors and their supporters. SNRIG does not seem to serve as a guardian of editorial integrity, upon which authors can rely to protect them from unjustified retractions. Rather, the mandate of SNRIG seems to be providing support to editors, and, if need be, to guide them safely through an external review of their practices. Why else would they not react to the information repeatedly provided to them about this case?

3.4 Lessons on editorial practices and competing interests

In this case, the journal behaves as if the code of conduct for maintaining research integrity, ethical standards, and guidelines apply to the authors, but not to them. The editor disrespected the authors and ignored significant dissent even from members of his own editorial board (Srholec 2021; Abramo et al. 2022). The journal doubled down on the hypocrisy by replying to the criticism that ‘in our opinion, it is important to see retractions as a last resort, to be used only if other means of correction cannot be applied’ (Zhang 2022: 1), which is the exact opposite of what happened here. The journal refuses to acknowledge any mistake, not even the factual error in the retraction notice, which itself begs for correction. But even this bare minimum of what a scientific journal should feel obliged to do after being alerted to its own error has not been forthcoming, and the journal does not signal any willingness to address the situation further.⁸

Another disturbing example of a double standard is how the rule for disclosing a conflict of interest has been bent in this case. Springer Nature has aligned business interests with the complainant, the major open access publisher that triggered the retraction, through a joint ultimate owner (see, e.g. the Economist 2013; Nature 2013; Science 2021). Nevertheless, the journal failed to report this competing interest throughout its handling of this case.

It might well have been that the editor was not aware of this connection before the retraction, which is why it was not disclosed in the retraction notice. But it was certainly brought to his attention during the COPE review, at the latest. When COPE enquired about this, the editor downplayed the issue by claiming that ‘[t]he Springer Nature teams (editorial and the research integrity group) supporting the Editor-in-Chief have not had conversations with Holtzbrinck [the joint owner] about this—there is no conflict of interest. Concerns regarding business interests would never be a criterion for investigating an article.’ The COPE review went along with this answer by concluding that ‘with regard to the concerns about competing interests, the editor indicated that the journals’ follow up focused on scientific items and not on legal or business interests’ (COPE 2022: 3–4) and with that, closed this line of enquiry.

Overall, this is not consistent with generally understood principles for disclosing a conflict of interest. COPE itself—with a reference to Wikipedia—specifies that ‘the presence of a conflict of interest is independent of the occurrence of impropriety’ (COPE 2015: 1). The same Wikipedia page further explains that ‘the existence of such conflicts is an objective fact, not a state of mind… a conflict of interest exists if the circumstances are reasonably believed… to create a risk that a decision may be unduly influenced by other, secondary interests, and not on whether a particular individual is actually influenced by a secondary interest’ (Wikipedia 2023). Finally, Springer Nature stipulates—in a section devoted to authors—that ‘disclosure of interests provides a complete and transparent process and helps readers form their own judgments of potential bias.’ (Springer Nature 2023b).

From this, it follows that a competing business interest should have been disclosed regardless of whether it influenced anything in practice, and regardless of how the potentially affected parties feel about it. What matters in this situation is that readers should be informed so that they can make up their own minds about it, not whether those who are implicated deem it to be relevant for readers to know. Surely, the editor would not easily let an author who fails to disclose being affiliated to someone with competing interests go unchallenged, just because she claims not to have had a conversation with that person about a published work. Failure to declare a conflict of interest is considered to be serious academic misconduct and, as explained above, a legitimate ground to retract published content. Yet neither the editor, the journal, or the publisher—in an agreement with COPE—feels compelled to amend the retraction notice in this case with acknowledgment of this problem. In fact, the retraction notice should probably be retracted based on this defect alone.

It should be noted that just before the retraction notice came out the editor offered to the authors to submit a new version of the paper that is reworked under his guidance for peer review to replace the previously published version in the same journal. The authors did not use this offer, because they did not understand what is wrong about the paper and how it could be repaired in the first place, despite repeatedly asking the editor for the explanation, and because that would imply admitting guilt of their side. It should not come as a surprise at this point that, when further pressed for answers, the editor closed the conversation with the authors by reminding them that there is the possibility to file a complaint with COPE. All attempts to seek a remedy therefore eventually led to the futile COPE review.

4. Reflections and recommendations: How to revamp the system of retractions?

As illustrated by the case presented above, there are distressing signs that the system of scientific publishing may lack efficient mechanisms to ensure that journals and publishers follow generally accepted retraction guidelines and, by corollary, remedy unjustified retractions. The fact that the authors and their supporters in this case repeatedly called for justice at various levels to no avail indicates that this cannot easily be blamed on the failure of a particular person or a mechanism, but that there is a deeper problem that needs to be addressed. Authors whose rights have been trampled by an unscrupulous editor may have nowhere to turn for a fair resolution. What should be done to improve this situation?

COPE’s capacity to facilitate the resolution of disputes over retractions has proven to be rather rusty. First and foremost, therefore, COPE needs to be equipped to investigate author’s complaints regarding retractions. COPE needs the resources to ascertain what has really happened during a retraction and whether there are credible reasons to doubt the position of the defendants, rather than to groundlessly side with the editor and publisher. Importantly, COPE should not be at liberty to cherry-pick which aspects of its guidelines are enforced and which can be overlooked in any given case. In particular, COPE needs to be ready to determine whether a retraction was justified on its merits, which is admittedly the key aspect of the whole procedure, rather than falling back on reviewing formalities.

Second, COPE needs to become independent of publishers by more than just proclamation. Until COPE secures a different source of income than funding from major publishers, which, as noted above, puts it into a conflict of interest with regards to them, it should stop pretending to the research community that it is in a position to enforce its retraction guidelines vis-à-vis their journals. Authors should not fool themselves by assuming that there is an independent institution within the system of science publishing that will conduct a fair and objective review of a disputed retraction. The case discussed in this paper provides a testimony to the fact that such a mechanism may not exist today, if it were ever there in reality.

If COPE does not upgrade, a different organization should take over facilitation of the resolution of disputed retractions, and perhaps other issues in science publishing as well. In fact, it is alarming that there is not an independent body, akin to a consumer protection association, to guard the interests of authors and thus serve as a counterweight to publishers, especially as authors increasingly find themselves directly in the position of customers paying for the services of journals in the context of open access publishing. The absence of such an organization is becoming even more troubling with the emergence of open access publishers that use aggressive and indiscriminate practices not only to solicit contributions to their journals, but also to suppress feedback on their services.

At the publisher level, Springer Nature should improve its internal mechanisms to check whether disputed retractions are fully justified, and whether due process has been followed, particularly if there is a significant backlash against the retraction from the research community, signalling that indeed something might have gone wrong. SNRIG seems the natural entity that should be endowed, like an ombudsman, with the resources and authority to investigate disputed retractions internally, rather than to conveniently refer complaints for review by an external organization. Most importantly, SNRIG should be in a position to hold editors accountable for any missteps. Of course, this would cost more money, but there seems to be a trade-off between cost savings and quality, and the correct balance seems not to have been reached at this time. Other publishers might arguably also benefit from an audit of these procedures under their own roofs.

One mechanism that helps to expose misconduct from within is a whistleblowing channel.⁹ This could be in synergy with other agendas of the aforementioned consumer (author) protection association, or it could be a watchdog agency established by universities, which would be money well spent, considering how much universities pay to publishers in access fees. Ideally, publishers should maintain their own channels, but in the context of this paper that seems wishful thinking, particularly for publishers struggling to get even the basics right. Nevertheless, this would go a long way to bolster the trust of stakeholders that third-party interests do not influence editorial decisions.

As far as retraction guidelines are concerned, the case presented above shows that there is a need for more transparency of retraction procedures. If an editor decides to solicit the opinions of other experts to guide her decision, the post-publication peer review should be open by default. The editor should be obliged to provide authors with full texts of the review reports and all other information that was used to reach the decision to retract their paper. The post-publication peer review reports should be published as an attachment to the retraction notice. Withholding anything, except what is necessary to protect the confidentiality of the reviewers, should be considered unacceptable.

5. Concluding remarks

Upholding high standards of research and editorial integrity is essential in the production of scientific knowledge. Integrity is what makes the difference between simply crunching numbers and actually engaging in science, and between what appears in vanity press and science publishing. Scientists have the responsibility—personally and collectively—to ensure that editorial principles are not degraded to empty phrases in obscure sections of journal webpages, or that they can be twisted or overlooked when convenient. It can be futile to complain about editorial missteps to those who either directly bear the blame, or to their associates. One needs to be able to turn elsewhere—to an independent arbiter—for fair treatment.

However, at least when it comes to retractions, this paper shows that, in the system of scientific publishing, there may not be a reliable independent body with the capacity to resolve complex disputes. It is disconcerting that a paper like Macháček and Srholec (2021) was retracted without any prove of misconduct or wrongful application of methods, that the journal and publisher has swept this incident under the carpet, and that the editor has evaded accountability. Against this backdrop, it is even more troubling that attempts by the authors and their supporters in the research community to seek a remedy led nowhere, because the existing complaint mechanisms are insufficient to meet the need. This sets a dangerous precedent that might encourage business and other third-party interests to suppress inconvenient research by retractions.

Sadly, the case in point discussed in this paper is not an isolated incident. Already, there are other cases that provide valuable context, including the developments around the article that found journals belonging to a major open access publisher to have predatory characteristics (Oviedo-García 2021; Retraction Watch 2023b), a major open access publisher rejecting an editorial that criticized the peer review process in its journal for publication (Horbach, Ochsner and Kaltenbrunner 2022), and a noteworthy expulsion of one of its founders from COPE after the founder dared to complain about a possible editorial problem in a leading science journal (Wilmshurst 2022). Clearly, there may be more cases of this kind. It is important to learn from them and to upgrade the associated processes, particularly the protection of authors, but also to bolster protections for reviewers and other stakeholders who are in far more precarious positions than they may realize, to ensure that missteps are minimized and to make the system more resilient.

In hindsight, perhaps the most sobering aspect of the retraction discussed here is that the true reason the paper attracted the ire of the open access publisher, as was highlighted in its letter that triggered the retraction (Retraction Watch 2021), was not necessarily anything written in the paper itself, but a news feed about it that resonated in social media (Singh Chawla 2021). After all, as already noted above, a number of similar papers using the same source of data in essentially the same way as in the contested paper have been published in highly respected journals, including the same journal, but no-one has attempted to retract them (for more elaboration on this see online Supplementary File S1 and Srholec 2021). If not for the social media frenzy, which stepped on the toes of the complainant, the paper would have most likely never been marked for retraction.

One glimpse of light at the end of the tunnel is the fact that the paper was eventually republished, after only minor revision that did not address any of the alleged flaws used to justify the retraction, in the journal Quantitative Science Studies (Macháček and Srholec 2022b), along with an editorial that encourages readers to reignite the discussion (Waltman and Larivière 2022) that others tried to close. Clearly, there are academics in the field of scientometric studies, including also signatories of the letter by Abramo et al. (2022), who do not hesitate to stand up against injustice and risk appearing in the crosshairs of the aforementioned interests themselves, for which they deserve great respect. Nevertheless, this does not address the broader systemic problem that the retraction exposed, which this paper hopefully helps draw attention to.

Notes

1

An earlier version of this letter—signed by 36 of members of the Distinguished Reviewers Board of Scientometrics and/or recipients of its Derek de Solla Price Medal—was sent to the journal and to the authors shortly after the retraction.

2

It is difficult to understand—or at least this was never explained to the authors of the paper or Abramo et al. (2022)—what the editor means by referring to Scopus in this context and by concluding that this database cannot be considered to be a control group. Nor did the editor explain what (and how) a control group should have been used in this econometric analysis.

3

Abramo et al. (2022: 2) confirm that to retract the paper for the lack of a control group ‘is puzzling. The use of control groups in scientometric analyses is uncommon. Control groups are typically used in experimental studies. Most scientometric studies, including the one by Macháček and Srholec, are of an observational nature. Observational studies typically use control variables in regression analyses to control for confounding factors, and this approach was also taken by Macháček and Srholec.’ The authors made a similar argument to the editor in their reply to comments from the post-publication peer review (see Supplementary File S2).

4

As explained in the paper, the languages spoken in a country are considered because the data sources—the use of which is widespread in this literature—predominantly cover journals in English, which is likely to entail a bias that needs to be taken into account when interpreting the cross-country differences.

5

Abramo et al. (2022: 2) alert the editor about the error in the retraction notice as follows: ‘The second reason is factually incorrect. The analysis of Macháček and Srholec does not consider the language of publications. It considers the language of the countries in which authors are located.’ The authors point to the fact that the editor’s argument about languages is confused and invalid in an email to the editorial board and the editor even before the flawed retraction notice was published online (Srholec 2021). This point was subsequently reiterated in comments under the first post about this case by Retraction Watch (2021).

6

It should be emphasized that the editor never presented any evidence that the findings are unreliable. He only speculates that this could be the case. Even before the retraction notice was published online, the editor was notified that the main results are robust to these two points, ie, even if they were relevant, the main conclusions of the paper remain intact, and any inconsistencies along these lines, if substantiated, could be satisfactorily addressed by a comment or a correction. Nevertheless, he retracted the paper anyway.

7

All documentation underlying the COPE review, including the full text of the ruling, is available upon request from the author and for download from Srholec (2021).

8

In early 2023, I offered a comment to the Co-Editor-in-Chief for publication in Scientometrics, which among other items urged the journal to issue a correction of the retraction notice with regards to the factual error about languages, but there was no reply from her. Evidently no one else felt compelled to request a correction.

9

I thank one of the reviewers for pointing this out.

Supplementary data

Supplementary data are available at Research Evaluation Journal online.

Acknowledgements

I thank the editor and two anonymous reviewers for their valuable comments. Any ambiguities, omissions or errors are the authors' responsibility.

Funding

Financial support from the Czech Academy of Sciences for the R&D&I Analytical Centre (RaDIAC) is gratefully acknowledged.

Conflict of interest statement. M.S. is the author of Macháček and Srholec (2021).