Predictive value of Altmetric score on citation rates and bibliometric impact

Abstract Background Bibliometric and Altmetric analyses provide different perspectives regarding research impact. This study aimed to determine whether Altmetric score was associated with citation rate independent of established bibliometrics. Methods Citations related to a previous cohort of 100 most cited articles in surgery were collected and a 3-year interval citation gain calculated. Citation count, citation rate index, Altmetric score, 5-year impact factor, and Oxford Centre for Evidence-Based Medicine levels were used to estimate citation rate prospect. Results The median interval citation gain was 161 (i.q.r. 83–281); 74 and 62 articles had an increase in citation rate index (median increase 2.8 (i.q.r. –0.1 to 7.7)) and Altmetric score (median increase 3 (0–4)) respectively. Receiver operating characteristic (ROC) curve analysis revealed that citation rate index (area under the curve (AUC) 0.86, 95 per cent c.i. 0.79 to 0.93; P < 0.001) and Altmetric score (AUC 0.65, 0.55 to 0.76; P = 0.008) were associated with higher interval citation gain. An Altmetric score critical threshold of 2 or more was associated with a better interval citation gain when dichotomized at the interval citation gain median (odds ratio (OR) 4.94, 95 per cent c.i. 1.99 to 12.26; P = 0.001) or upper quartile (OR 4.13, 1.60 to 10.66; P = 0.003). Multivariable analysis revealed only citation rate index to be independently associated with interval citation gain when dichotomized at the median (OR 18.22, 6.70 to 49.55; P < 0.001) or upper quartile (OR 19.30, 4.23 to 88.15; P < 0.001). Conclusion Citation rate index and Altmetric score appear to be important predictors of interval citation gain, and better at predicting future citations than the historical and established impact factor and Oxford Centre for Evidence-Based Medicine quality descriptors.


Introduction
Academic reach and impact are sought after by author and journal editorial boards. Currently, the number of citations achieved by any given article is the most commonly used measure of academic reach, and the foundation of author Hirsch index (h-index) and journal impact factor (IF). In many countries, higher education institutions allocate funding according to academic outputs associated with journals held in high esteem on the basis of these traditional measures 1,2 .
Citations, however, take time to accumulate and other faster means of assessment have risen: alternative metrics, or Altmetrics. Altmetrics extend the concept of citation beyond mention in other scholarly articles by recording how often an article is downloaded, and applies to people, journals, books, data sets, presentations, videos, source code repositories, and web pages. The Altmetric score (AS) calculates impact based on diverse online research outputs, including social media, blogs, online news media, and online reference managers 3 . Consequently, the concept of impact remains as important and relevant a philosophical question as ever 3,4 .
Articles with a higher AS have been reported to be associated with higher citation counts 3,5 , but whether this symbolizes preliminary increased curiosity or sustained citation growth is unknown. The aim of this study was to identify the factors associated with citation accumulation, using a historical control cohort of the 100 most cited articles in surgery (2018) 3 . The hypothesis was that AS is directly related to prospective citations and an indicator of citation trajectory.

Methods
Citation counts related to a previously reported cohort of 100 alltime most cited articles from a search performed in 2017 associated with the keyword 'surgery' were reanalysed 3 . Thomas Reuters' Web of Science citation index was accessed again, providing a 3-year interval citation gain (iCG). Previously collected bibliometric data, including AS, citation count, citation rate index (CRI; total number of citations divided by number of years since article was published) 6 , 5-year IF, and Oxford Centre for Evidence-Based Medicine (OCEBM) 7 quality of evidence levels were used to classify citation rate prediction.

Altmetric scores
An AS is a weighted count of all of the mentions that Altmetric has tracked for an individual research output, and is designed to be an indicator of the degree of notice or attention an item has received. Scores are accumulated in an adjusted fashion to reflect the relative influence of different sources. For example, getting newspaper attention is more difficult than Twitter attention from a single individual; consequently, newspaper articles are more likely to receive attention than tweets and are thus allocated a higher contribution to a research output's score. AS values are always whole numbers and, importantly, only the first mention from a source contributes to its accumulation. Some sources of attention will contribute less than 1 whole point to an output's AS; in such instances, the AS is rounded up to the next whole number. The weighted scoring system can be found on the Altmetric V R website 8 .

Web of Science category and quartile rank status
A journal's IF category rank was recorded from Web of Science. A percentile rank (z) was calculated by dividing the rank (x) by the number of journals in a subject field (y), where z ¼ x/y 9 . The percentile rank is categorized into four quartiles (Q1-4), with the most prestigious journals in the top quartile. Q1 journals are those belonging to the top 25 per cent, and Q4 are those within the bottom 25 per cent in a given subject field.

Statistical analysis
All data were expressed as median (i.q.r.) and non-parametric statistical methods used throughout. The OCEBM scoring system for quality of evidence was reported in a Likert-scale format (level 1, highest level of evidence-systematic review or meta-analysis of RCTs; level 5, lowest level of evidence-expert opinion without critical appraisal) 7 . The Spearman correlation coefficient test was used to test the relationship between quality of evidence and iCG. The iCG was clustered into deciles to meet the test assumptions. Receiver operating characteristic (ROC) curve analysis was used to assessed the predictive value of continuous variables, with the primary outcome measure iCG dichotomized at the median and top quartile 10 . Univariable and multivariable logistic regression models were developed to identify independent associations with the primary outcome measures. Variables with P < 0.100 in univariable analysis were included in the multivariable analysis, using a backward conditional model. All statistical analysis was performed using SPSS V R version 25.0 (IBM, Armonk, New York, USA).

Results
The total cumulative citations and iCG for all articles to date were 100 127 and 28 990 respectively. Ninety-eight articles were published in Q1 journals, one in a Q2 and one in a Q3 journal; there were no articles published in Q4 journals. Every article received more citations, with a median iCG of 161 (i.q.r. 83-281). Seventy-four articles had an increase in CRI (median gain 2.8 (-0.1 to 7.7)), it remained constant for one, and decreased for 25. AS increased for 62 articles (median gain 3 (0-4)), did not change for 36, and decreased for two. The top 10 articles related to the highest iCG can be found in Table 1 [11][12][13][14][15][16][17][18][19][20] , and the full article list is available in Table S1. Of the original 10 most cited articles in 2017, eight maintained their top ten status 3 years later in terms of total citations.
Univariable and multivariable analysis of factors associated with above median and upper quartile of interval citation gain Univariable and multivariable analyses of factors associated with iCG are shown in Table 3. In univariable analysis, multicentre studies (P ¼ 0.023), articles with first or last author featured more than once in the 100 most cited articles (P ¼ 0.044), an original AS of at least 2 (P ¼ 0.001), original citation count 521 or more (P ¼ 0.008), or original CRI at least 34.4 (P < 0.001) were associated with higher iCG dichotomized around the median. When dichotomized around the upper quartile value, articles where the first or last author featured more than once (P ¼ 0.041), those with an original AS of at least 2 (P ¼ 0.003), original citation count 521 or more (P ¼ 0.048), or original CRI at least 34.4 (P < 0.001) were associated with higher iCG. In multivariable analysis, only CRI was associated with higher citation accrual when iCG was dichotomized around the median (odds ratio (OR) 18

Discussion
The principal finding of this study was that articles that amass citations at faster rates continue to grow their citations in an autonomous fashion that reflects the type of study, quality of evidence, affiliation of author or academic productivity. Articles with a CRI of at least 34.4 were 18 times more likely to accrue greater numbers of citations. AS was associated with a higher iCG; univariable analysis revealed that an AS of 2 or more conferred a fivefold multiplication factor with respect to citation accrual over a 3-year interval, but this was not verified by multivariable analysis.
Despite the widespread use of several impact metrics to gauge professional success, it remains unclear just how technological or scientific power evolves. Sinatra and colleagues 21 quantified the changes in impact and productivity of an academic scientist throughout a career, and reported that impact, as measured by influential publications, was distributed randomly through any given sequence of publications. This random-impact rule allows the formulation of a stochastic model that disengages the effects of individual ability, productivity, and chance, implying the existence of universal patterns governing escalation of academic productivity. The model assigns a unique individual and stable parameter Q to each scientist, which accurately predicts the evolution of impact, from the h-index to cumulative citations, and independent acknowledgements such as prizes. This suggests that a scientist can influence an article's impact, regardless of content, which is supported by the observation that authors with more than one entry in the 100 most cited list went on to accrue more interval citations, irrespective of academic affiliation [9][10][11][12]15,[20][21][22][23][24][25][26][27][28][29][30][31][32] . Having multiple publications was not autonomous of CRI in predicting iCG on statistical modelling, supporting a close association between author influence and citation accrual. Certain authors, or teams of authors, appeared to be associated with a positive multiplication coefficient or constant which, when allied to a novel project, good fit, and timeliness, accelerated and boosted citation trajectory.
Wang and co-workers 33 reported an arithmetically derived mechanical model which showed that ultimate article impact, characterized by total lifetime citations, was largely predicted by a single factor described as fitness. Fitness is the perceived novelty and relevance that an article attracts from the research community at large. This may explain why the Clavien-Dindo classification article 11 , which received the greatest number of citations and had the highest iCG in this study, was one of the highest ranking articles in bibliometric analyses of surgery 34 , visceral surgery 35 , and laparoscopic surgery 36 . When Wang et al. 33 compared articles from journals with an impact factor of 3.26, 10.48, and 33.62, articles with similar fitness received initial citations in a journal-dependent manner, but ultimately received similar lifetime citation counts, which may help to clarify why journal IF did not predict citation count in the original 2018 study 3 , or iCG in the present study. The findings described by Wang and colleagues justify how proximity of citation gain, which augments journal IF, is supported by social media platforms. Whether the impact of social media simply shifts mean citation count or boosts total citation count and impact is unclear 33 . With increasing competition for research funding, emphasis has migrated towards proving impact using bibliometric profiles and, historically, article citation count has always been synonymous with impact. Bibliometric use is supported by the present findings; in particular, CRI derived from citation count was associated independently with iCG. Academic reach is now facilitated by electronic devices, and journals have embraced this paradigm shift with a greater online presence. Zerrweck and colleagues 37 reported that 31.5 per cent (51.8 per cent recreational use) of general surgeons, and 39.7 per cent (68.6 per cent recreational use) of bariatric surgeons used social media on a daily basis for academic purposes, making it possible for researchers to play an active role in dissemination of their academic outputs, likely amplifying citations. Alternative metrics are part of this era in measuring impact. It remains to be seen whether preprint article repositories, which enable early sharing of articles on social media, result in higher AS and citations, providing journal editors with additional and early impact metrics.
This study has a number of inherent limitations. Articles included were published at different times and were therefore placed at different stages of citation accrual. Original AS values  were not recorded at identical follow-up times; nonetheless, AS and CRI were associated with a higher iCG, and were therefore irrespective of time lapsed. These bibliometric markers were strongly associated with citation accrual. All articles predate the founding of Altmetric scoring (2011) and the majority (89.0 per cent) predate the creation of Twitter. As such, conclusions regarding the impact of social media on AS cannot be established for this cohort, and future work should focus on prospectively collected serial measures of bibliometrics and AS over time using current articles. The 5-year IFs used in this study were acquired via the Web of Science database during the original data collection period in 2017. Some of these values may have fluctuated over time, having been higher or lower at the time of each paper's release. The 100 most cited articles related to surgery were the focus, and the findings must therefore be interpreted with caution, as these findings may not be applicable to scientific articles published in other research arenas, or articles published in lower Q-score-rated journals given the preponderance of Q1 journals in this cohort. Citations carry inherent bias in the form of institutional or self-citation that cannot be identified from an online database search. The study also has a number of strengths. Article citations and AS values were recorded simultaneously and from a reputable database-Web of Science. The bibliometric analysis has statistical power, limiting confounding variables, and bias relating to time from publication.  Despite the perceived reciprocal association and attraction between high-quality research articles and journals with a higher IF, it appears that article fit, social media promotion, and author power are also predictors of prospective article impact and citation accrual. If any journal's editorial strategy is to publish articles expected to acquire high citation numbers, early upload to one of the emerging repository websites may well develop as a preferred plan before official periodical submission. Such tactics would allow a period of attention, scrutiny, reflection, AS attainment, and early citation accrual. Such a strategy may be an improvement on the current peer review system before publication in a journal.

Funding
Royal College of Surgeons England Health Education and Improvement Wales