The impact of research in the Space Life and Physical Sciences (SLPS) is compared to that of similar ground-based research over a 20-year period. As SLPS publications are cited initially much lower than similar ones of ground-based research, use of a short-citation window (1–4 years) is detrimental in impact assessment of SLPS in comparison with ground-based research. The citation peak of space research is reached much later than that of ground-based research. During the first 10 years after publication of space research, its field-normalized impact is way behind comparable ground-based research, but it catches up during the next 10 years, although not completely. According to experts, the intermittent character of SLPS research may have contributed to this pattern. Bibliometric indicators should be calibrated to provide optimal monitoring of the citation impact of SLPS research, and of topics that face similar conditions.
About half a century ago, research efforts started in the space life and physical sciences (SLPS), relating to effects of space conditions, such as microgravity and radiation, on biological and physical systems. In the life sciences, research has focused on human, animal and plant physiology and space medicine, radiation and gravitation biology, and, more recently, exobiology. In the physical sciences, crystal growth and materials sciences, combustion processes and fluid dynamics have been studied, with an increasing importance of fundamental physics. Space research is not a formal research subfield in its own right, and few scholarly journals are dedicated exclusively to space life sciences (SLS) or space physical sciences (SPS).
Experiments in SLPS tend to be relatively costly compared with ground-based research (which limits the extent of the research for financial reasons), are technologically complex, and are subject to the uncertain availability of suitable space missions. The ‘boom or bust’ nature of space exploration as evidenced by the Challenger and Columbia flight incidents is likely to influence the pattern of publications on space experiments. Successful space flights with many experiments are likely to boost publication output, whereas failed space flights yield nor experimental data nor publications. SLPS publications may contain references to earlier SLPS papers, and therefore a drop in SLPS publications in turn might cause a decline in citations to previously published SLPS work.
In this article, we examine the validity of methodological assumptions behind bibliometric studies in the SLPS and evaluate the viability of bibliometric data collection methods, monitoring tools, and indicators in that context. The basic point addressed by the study is: to what extent, if at all, can we monitor the citation impact of SLPS research. An important issue in SLPS is whether space LPS publications need more time than ground-based LPS publications to collect citations. It is conceivable that the intermittent nature of spaceflight experiments might have a negative effect on (short-term) impact, while diffusion of SLPS results to ground-based science might also require more time than the ground-based science average, as SLPS experimental conditions are much more difficult to control than those of ground-based research. Furthermore, we study ageing effects of the SLPS literature.
Many studies have analysed the adequacy of citation windows in relation to peak moments in the citation history of scientific literature, and particularly in relation to the moment of obsolescence of scientific literature (Nederhof et al. 1993; Glaenzel and Schoepflin 1995; Moed et al. 1995; Larivière et al. 2008). Nederhof (2006) compared the length of citation windows in various fields, and concluded that in science these usually peak or reach a level close to the highest score in the 3rd year after publication. Studies on the adequacy of applied citation windows underlying journal impact factors have shown that in most fields in the natural and engineering sciences, the moment of the citation peak reached by published material lies somewhere around the 3rd–4th year after publication (van Leeuwen et al. 1999). However, in some social sciences and humanities, these peak moments seem to be even farther away from the time of publishing (van Leeuwen 2006).
Other research focused on various other aspects of the scientific communication process, such as the adequacy of applied citation windows, particularly in relation with journal impact measures (Moed et al. 1998), the effect of publication delays on citation patterns of fields and journals (Luwel and Moed 1998), late recognition (Braun et al. 2010), and more general patterns causing scientific literature to become obsolete (Costas et al. 2010, 2011). On the whole, citations obtained in early years are found to be a good predictor for those obtained in later years (Adams 2005; van Dalen and Henkens 2005).
2. Data collection
To obtain an optimal coverage of SLPS publications worldwide during the period 1985–2004, we retrieved publications from five sources:
INSPEC (SPS) zero-gravity-experiments, 1985–2004: P = 1364;
MEDLINE/PUBMED NASA files (SLS), 1985–2004: P = 1653;
ESA Erasmus Experiment Archive, 1985–2004: P = 431;
ESA Peer reviewed SLS publications up to 1999 (SLS), 1985–99: P = 562;
Thomson Reuters ISI Citation Indices (WoS), 1997–2004: P = 2900).
Publications relating to the SPS were retrieved from INSPEC (1), a specialized database containing abstracts of physics-related publications. In INSPEC, the descriptor (DE) ‘zero-gravity-experiments’ was found to be helpful, particularly after removing papers dealing with instrumentation and space transport vehicles rather than space experiments. Also, papers dealing with ‘Volcanology’ were eliminated (here, microgravity effects occur outside the scope of the present study).
In addition, the medical publications database PUBMED (2) was searched for SLS publications. Here, two specialized publication files from NASA were retrieved. For publications on space life sciences, we searched PUBMED files (PUBMED: http://spaceline.usuhs.mil/sampleSearches.html; listed September 2005). Here, the NASA had posted two specialized publication files. The main publication file covers international publications from space flight experiments (‘Space flight experiment citations’) while a much smaller set of publications is called ‘moon/Mars/settlements/fractional gravity citations’ (listed September 2005).
Furthermore, the ESA Erasmus Experiment Archive (EEA) (3) contains descriptions of experiments funded or co-funded by ESA, both in space and in ground-based microgravity facilities and includes scientific publications in peer-reviewed journals. The EEA covers both publications from the SLS and the SPS. Only publications relating to ESA experiments conducted between 1985 and 2004 were retrieved. A second ESA file, ‘ESA publications up to 1999’ (4) contains world-wide publications in the SLS (ESA 1999).
Finally, publications were retrieved from the ISI Citation Indices (ISI) (5) by means of keyword searches. This effort was primarily intended as a back-up of the publication searches by means of the ESA, PUBMED NASA, and INSPEC publication databases. Since 1998, ISI extended its coverage of publications on CD ROM with the inclusion of publication abstracts. This is needed, as our searches, with few exceptions, tended to combine keywords related to microgravity or space conditions with content-related terms (e.g. ‘liquid bridge’). Searching titles as well as abstracts and keywords attached to the abstract either by the author(s) or the database considerably enhances the odds of successfully identifying SLPS publications. Given the expected poor yield of new additional SLPS publications in 1985–97, we focused our efforts on database years 1998–2004 (this also yielded some publications from 1997). SLPS experts contributed lists of relevant keywords or combinations of keywords, and screened retrieved papers.
Publications from INSPEC, PUBMED, and ISI were retrieved by means of keyword searches in titles, keywords, and abstracts of publications. Retrieved publications from the five sources were matched to the ISI database. We considered only papers classified in ISI as normal articles, letters, notes, and reviews, published in source journals processed for ISI on CD-ROM.
2.1.1 Definition of SLPS
Only publications were included that dealt with data obtained in experiments performed during spaceflight or in space-like microgravity conditions (airplane parabolic flights, sounding rockets, and drop towers), as well as preparatory research of space experiments. Also, publications were included that dealt with Astro/exobiology and human planetary exploration.
The present study relates to the SLPS publication output during the period 1985–2004. In the bibliometric analysis, citations were also collected during the period 1985–2004. Complete data on more recent articles were not available during the data collection period of this study. The study is based on a quantitative analysis of scientific articles published in journals and other serials (in brief: journals) processed for the CD-ROM versions of the Science Citation Index and eight associated indices (ISI): the Science Citation Index (SCI), the Social Science Citation Index (SSCI), and the Arts & Humanities Citation Index (A&HCI), extended with six specialty Citation Indices (Chemistry, Compumath, Materials Science, Biotechnology, Biochemistry & Biophysics, and Neuroscience).
2.2 Verification rounds
Papers retrieved from the five databases were screened by four SLPS experts. In a first round, retrieved ISI papers (N = 6,085 including duplicates) were screened if these did not occur also in the four INSPEC, PUBMED NASA, and ESA publication sets. A total of 4,222 ISI publications were not present in one or more of the four other data sets. About 2,200 papers were rejected in this screening, resulting in 2,002 new ISI publications or, after removing duplicates, 1,628 unique additional ISI publications.
A second verification round was conducted concerning publications with at least one ESA/European address. This resulted in the removal of 40 publications, mostly dealing with earth observations, non-relevant (space) engineering topics, satellite data, Mars Pathfinder and Mars Global Surveyor items, and astronomical topics. Finally, 17 ISI publications (9 SLS and 8 SPS) were added, based on institutional publication sets from two experts. As a result, the final data set of world SLPS publications counts 5,121 publications.
3. Bibliometric indicators
Field-normalized citation scores are computed. A reference value presents the mean citation rate of the subfields (journal categories) in which the research unit is active (FCSm, the mean Field Citation Score). Our definition of subfields is based on a classification of scientific journals into subject categories developed by Thomson Reuters (formerly ISI) for the Citation Indices. Although this classification is certainly not perfect, it is at present the only classification available to us. For each subfield, FCSm takes into account both the type of paper (e.g. normal article, review, and so on), and the specific years in which the research unit's papers were published. In most cases, a research unit is active in more than one subfield (i.e. journal category). In those cases, we calculate a weighted average value, the weights being determined by the total number of papers the research unit has published in each subfield.
The most important indicator CPP/FCSm compares the average number of citations to the oeuvre of a research unit (CPP) to an international reference value, namely the corresponding FCSm, by calculating the ratio of these scores. Self-citations are excluded in the calculation of the ratio CPP/FCSm, to prevent that the ratio is affected by divergent self-citation behaviour. If the ratio CPP/FCSm is above (below) 1.0, this means that the oeuvre of the research unit is cited more (less) frequently than an ‘average’ publication in the subfield(s) in which the research unit is active. The CPP/FCSm score is also designated as the ‘field-normalized’ citation impact. For a detailed description we refer to Moed et al. (1995).
In comparisons across year blocks (e.g. when publications from 1995 to 1999 are compared with those of another year block), it is important to focus on normalized indicators only, as these normalized values are free from influences by distribution and document types effects (Nederhof and Visser 2004).
4.1 Coverage of ISI Publications in SLPS
To gain insight in the ISI coverage of SLPS publications, references in the full set of SLPS ISI papers (1985–2004; N = 5,121) were matched with our extended ISI publication database (1980–2004). In this way, we can estimate the importance of ISI-publications to SLPS researchers by determining to what extent they themselves cite ISI CD-ROM papers, and to what extent other, non-ISI, documents. Due to the extension of our database, we could only trace references dated from 1980 onwards. Self-citations were included in this analysis, as we could not exclude all self-citations for non-ISI documents.
We discuss the main results for worldwide SLPS. The average SLPS paper contains about 27 references. On average, 17% of the references was dated before 1980. Of the references since 1980, 68.3% could be matched to ISI CD-ROM papers. This finding suggests that non-ISI documents are of limited importance to SLPS researchers, as they account for less than a third (31.7%) of the references in their papers. This contrasts strongly with the volume of non-ISI papers. In general, in SLPS, non-ISI papers outnumber ISI papers. For instance, in EEA about 70% of the publications is non-ISI. However, notwithstanding their much smaller volume of papers, the results indicate that the large majority (68.3%) of the citations is directed at SLPS ISI publications. The present findings support the use of ISI papers as the basis of a citation analysis. The citation analyses in this study include the publications that matter most to science in general (as indicated by the volume of citations) as well as cover the large majority of citations.
4.2 Sensitivity analysis of bibliometric indicators
To examine whether differences exist in the citation impact distribution between SLPS and ground-based research, we followed SLPS publications over time. In particular, the main bibliometric indicator, field-normalized citation impact, was traced.
For a first analysis, the period after publication during which citations are collected, the citation window, was varied from 2 years to 20 years, each time increasing the length of the citation window with 2 years. It should be noted that all analyses pertain to database years rather than publication years. Thus, for a publication from 1985, citations were first counted in 1985 and 1986 (2-year citation window), then in the period 1985–88 (4-year citation window), and finally during 1985–2004 (20-year citation window). For each publication year between 1985 and 2003, CPP/FCSm scores were computed for each of the 10 citation windows. For 2004, more data are needed than extant to compute even a 2-year citation window. For good measure, we have also included the results for about 50 SLPS publications stemming from 1984 that happened to be included during data collection.
Figure 1 shows the results. The findings indicate that in most cases, the field-normalized citation impact indicator increases over time. With either a 2-year or a 4-year window, field-normalized citation impact varies typically between about 0.20 and 0.70, and increases to a level between roughly 0.70–1 for citation windows of 16-year and longer. The only exception is formed by the 1997 publications, which are cited near world average initially, to decline somewhat subsequently. In our experience, the level of increase in CPP/FCSm witnessed in the bulk of publication year sets represents a rare occurrence. Usually, field-normalized citation scores change little between short-term and longer-term citation windows, and if there is a difference, it tends to be small. For ground-based research, CPP/FCSm is by definition stable at value 1 (but see below for an analysis that separates CPP and FCSm).
In Figure 2, data are combined for the publications produced during 1985–1994. This means that for citation windows up to ten years in length, averages are based on a constant set of publications. Of course, for longer citation windows, results depend on a smaller (and older) set of publications. Therefore, we can observe that for SLS publications, the field-normalized citation impact increases from 0.49 to 0.77 depending on the length of the citation window, while for SPS field-normalized citation impact varies between 0.56 and 0.94 (16-year window). The data points for the 18-year window are based on the publications from only 4 years with relatively small numbers, and are not homogeneous as regards citation impact. Therefore, the results for the 18-year window are insufficiently reliable. To cope with the problem that, for the longer citation windows, data come from a smaller number of years and depend increasingly upon the absolute impact level of individual years, a third analysis was made (Figure 3). Here, field-normalized citation impact values were normalized on those obtained from the 2-year citation window. Thus, results are only dependent upon the increase of the field-normalized citation impact score, and not on the absolute level of field-normalized citation impact. For instance, for publications from 1985, field-normalized citation impact was 0.30 for a 2-year citation window and 0.34 with a 4-year citation window. Dividing the 4-year citation window field-normalized citation impact score by that of the 2-year citation window score gives a value of 0.34/0.30 or 1.123. Data for all available publication years were added (in this case for 1985–2001; 2001 is the last year for which a 4-year citation window could be computed) and divided by the number of publication years available (in this case 17). Figure 3 shows that with an increasing citation window, the field-normalized citation impact about doubles. The doubling of field-normalized citation impact is reached somewhat faster for SPS (1–14-year citation window) than for SLS (1–16-year citation window). After that, further increase in field-normalized citation impact seems not lasting, at least not for SPS. More data are needed to determine reliably what happens when the citation window is further lengthened.
The above findings show a considerable increase in field-normalized citation impact with longer citation windows. But it is not clear what happens exactly: is CPP of SLPS increasing faster than that of FCSm (ground-based research; actually this includes a tiny bit of SLPS), is the one declining and the other stable over time, or is still something else the matter? To examine this more closely, a detailed analysis was made of the 1987 SLS publications (N = 47) (Figure 4). Here, citations were counted over subsequent 2-year publication periods: first 1987–88 (citation window 1–2 years), then 1989–90 (citation window 3–4 years) and so on to 2003–4 (citation window 17–18 years).
The results for the ground-based Life Sciences research, as embodied in the FCSm score, show a pattern familiar from many bibliometric studies (e.g. Garfield, 1979): few citations are received during the first 2 years after publication (FCSm 1–2 years = about 1), but the citation rate peaks during the 3rd and 4th year after publication. In subsequent 2-year periods, citation impact gradually drops until, in the 17th and 18th year, the citation rate is less than half that of the peak years (3rd–4th year). However, the pattern of SLS is quite different. During the first 2 years after publication, citation rate is only half that of ground-based research, a statistically significant result. The first citation peak of SLS is reached only in the 5th and 6th year, 2 years later than ground-based research.
Remarkably, during the first 10 years, the citation impact of SLS remains below that of FCSm, although the difference is largest in the first 4 years. However, during the entire second decade, SLS impact exceeds that of the sagging ground-based research. Due to the limited number of publications involved, the apparent second citation peak of SLS at the start of the second decade after publication (Years 11–12) may be incidental, particularly as it is preceded by a minor slump in the previous 2 years (Years 9–10).
From a SLPS perspective, experts considered that the special circumstances surrounding the conduct of this type of research in space play an ‘overwhelming role’ in making the time to collect citations in this area far longer than in comparable ground-based work. These circumstances were cited as:
The time between the conception of an experiment/research activity and space flight is very long (one expert estimated that it takes from 5 to 7 years) due to both a shortage of flight opportunities leading to long queues and long development times for space experimental hardware. It takes another 5–7 years to do the follow-on experiments; the resulting publication can refer to earlier work only then.
An exacerbating factor for Europeans is that usually there is no national funding of a project until ESA has allotted a firm flight opportunity, and this can introduce further delays.
A secondary effect is that many researchers who do space research are also active in ground research that has faster dynamics. So space research gets regarded as a long-term affair with little pressure to publish and cite at a high rate.
There may be an effect of ‘hot’ research areas on ground research, which are highly cited, and which are not represented in space research, such as, for example in the life sciences, those employing the techniques of molecular biology, genetics, and the like. The papers in these ‘hot’ areas survive less long, i.e. are cited for not very long, since the very active field moves along fast and produces publications that ‘outdate’ the ones recently published.
On their own, none of the publication retrieval methods managed to provide a good coverage of SLPS publications. The study shows that sole reliance on keyword searches in SCI and related databases does not ensure an optimal coverage of publications and that it is necessary to combine several methods of publication retrieval as well as to combine results from several publication databases. Even then, verification of retrieved publications by experts was shown to be very important. Nevertheless, the SCI searches alone accounted for the single greatest yield of valid publications.
Assumptions behind bibliometric indicators and bibliometric data collection were critically examined. Standard bibliometric methods fail to gauge the citation impact of SLPS research and give rise to misleading conclusions about research performance. An examination of the optimal citation window for bibliometric monitoring showed that SLPS is by no means a standard science research field. SLPS citation impact does not peak, as in many science fields, after 3–4 years, but requires the use of long-term citation windows. Although strong requirements for accountability in SLPS research may have contributed to the production of large volumes of non-serial publications and grey literature, an analysis of the percentage of references in SLPS articles in scientific journals to ISI source journals provides clear evidence of the importance of publication in peer-reviewed scientific journals in SLPS research. Accurate calibration of bibliometric monitoring tools is necessary in order to obtain valid results, and to allow the formulation of relevant policy conclusions on research performance on various levels of aggregation and time.
Although the analysis that compared the impact (CPP) of SLS with that of ground-based research (FCSm) is based on a limited number of SLS publications, the findings confirm the results of the previous analyses: use of a very short-citation window (1–4 years) is detrimental in impact assessment of SLS and SPS, as SLS and SPS publications are cited initially much lower than similar ones of ground-based research. As years progress, citation rates of SLPS and ground-based research converge, and in later years, citation impact of SLPS publications declines more slowly than those of ground-based research.
Use of a very long fixed citation window would limit bibliometric analysis to rather old research, which is not very informative for policy purposes. For optimal short-term monitoring of SLPS it is recommended to exclude the first years after publication (at least the first 2 years and perhaps even the first 4 years) from the citation window. Therefore, particularly in comparisons with ground-based research, we recommend to follow citations between the 5th–6th years or 5th–8th years after publication (depending on policy needs and the availability of sufficient citation data). In addition, long-term indicators should be used in comparing SLPS with ground-based research. For comparisons within SLPS, and particularly within SPS or within SLS, the length of the citation window is of less importance, as all publications that are monitored face the same conditions. This is particularly true for field-normalized citation benchmarks.
The findings indicate that SLPS research tends to have an impact below that of ground-based research, even with a long citation window, although an even longer time may be needed to evaluate this more fully. In future research, we intend to examine this finding more closely.
We expect that the present findings are to some extent exceptional in that research will not be often dependent on conditions that are as intermittent as those relying on (manned) space flights. However, there may be many other topics in which research is dependent upon rare special expeditions, more or less unique equipment that produces results at a highly intermittent rate, and so on. Research on such topics might display some of the characteristics found in the present study, and care needs to be taken that appropriate citation windows are selected in bibliometric monitoring studies.
The study was supported by ESA under ESTEC contract No.: 19170/05/NL/VJ. We thank Anthony van Raan for his comments and support, and Nick Larter for his administrative support. We also thank Ed Noijons and Renald Buter for their contributions, and in particular the members of the 4CON Science team, Dr Berndt Feurbacher, Dr G. Horneck, Dr Jean Claude Legros, and Prof. Peter Scheid.