Publishing in top-ranking journals in the social sciences and international relations requires writing with clarity. Accurately described and transparent methods sections ensure high-quality academic writing. The methodology section of empirical papers should explain the exact steps taken by the authors when operationalizing concepts and testing hypotheses to facilitate replication. This also allows for monitoring quality, challenging findings, and promoting good scientific practices. The quality of methodology sections is the result of the interaction between academic cultures of data sharing, effective application of rules, and good-quality research data management (RDM). This article evaluates the impact of standards on replicability. We present an empirical analysis of a set of sixty-six articles published during the period 1984–2013 that use data from all waves of the European Values Survey. We find differences demonstrating the impact of good RDM and data policies on good scientific practice.

The focus of this article is the impact of standards on methods sections in social science and international relations (IR) journal articles. The methods section describes the research procedure, translating theory into testable hypotheses with information on the research design, hypothesis, operationalization of measures, units of observation, data collection, verification, and analysis. The methods section should provide sufficient information to allow other researchers to replicate the research without requesting extra information from the author (Berg 1998).

This article has three goals. First, we distinguish between papers using primary and secondary data and formulate a typology of what should be included. Second, we examine current data policies, making associations regarding the decisions and actions of research stakeholders and the effect on good research data management (RDM) practice as reflected in published articles. Third, we test for differences over time and between data types using datasets from the four waves of the European Values Study (EVS). Our data come from coding a set of sixty-six articles based on EVS data published between 1984 and 2013. This ensures variation in RDM quality across waves. We find differences that demonstrate that data policies and RDM expectations have made a difference in the replication quality of articles.

An impulse behind the emergence of behaviorism in the social sciences, including IR, was to bring scientific methods and rigor to social inquiry. With scientific methods come the requirements to replicate, recreate, repeat, revise, or recompute methodologies and data to test the accuracy and validity of claims purportedly based on evidence. As King (1995, 444) states: “the only way to understand and evaluate an empirical analysis fully, is to know the exact process by which the data were generated and the analysis produced.”

Scientific replication—the ability to independently verify the results of an experiment through copying the method of the original experiment or data analysis—goes back to Geber’s investigations into alchemy in the eighth century. Easton (1953) declared the ability to verify one’s generalizations as a foundation of behavioral political science. It is unsurprising in an age of greater data availability that academic publications are using data more frequently than in the past (Busenitz et al. 2003; King 2006). However, what is surprising is that the requirement to produce scientific outputs that can be independently understood through clearly described methods, and with the data made available to the research community within legal, commercial, and ethical boundaries, was for a long time an ideal rather than a norm.

Replicability can be taken as simply good scientific practice. Following Popper (1959, 45), we define it as: “Any empirical scientific statement can be presented (by describing experimental arrangements, etc.) in such a way that anyone who has learned the relevant technique can test it.” Science is the pursuit of a common goal: A deeper understanding of the world and the mechanisms that govern it. Enabling other researchers to replicate a study, to support or refute it, advances us toward that common goal (King 1995). Using replication standards in papers improves the quality of academic writing by bringing transparency to research. Good scientific practice requires allowing other scholars to reproduce the results or methods of the papers. A high-quality methods section should provide the map toward replication.

The 2003 Symposium on Replication in International Studies Research resulted in a set of papers—published in International Studies Perspectives—which broadly examined “issues concerning the potential replication of research results from a number of different conceptual and technical perspectives” (Boyer 2003, 72). Gleditsch, Metelis, and Strand (2003) reviewed data replication policies arguing that although they had been strengthened, journals failed to enforce them. However, the situation might be changing for the better (Gleditsch, Metelis, and Strand 2003). Ray and Valeriano (2003) studied barriers to replication by testing the replication policy of International Interactions from 1999 to 2001. They found existing policy was not properly implemented and better methods, procedures, customs, and disciplinary norms were needed. James (2003) looked at International Studies Quarterly replication policies and practices and concluded that IR and social sciences more generally were losing something important without replication policies. Russett (2003) suggested that not following the replication policy in the Journal of Conflict Resolution posed the risk of being professionally embarrassed and establishing a bad reputation for future publications with the journal. Gleditsch, Metelis, and Strand (2003) evaluated the impact of making replication data available in citations, with significant positive results. Bueno de Mesquita (2003) pointed out that replication policies could speed up manuscript reviews by providing better information for referees to make recommendations to editors. What is more, he argued, replication policies can significantly reduce errors in published research. Finally, King (2003) presented a technical solution, positing that persistent identifiers facilitate reliable sharing and citation of datasets, thereby enabling replication.

How to Write a Methods Section to Ensure Replicability

Although the aim of a methods section is to explain the research process and analysis procedure, there is a difference between papers using primary data and reusing data. This distinction leads to different parameters included in the methods section, and we acknowledge that distinction in this paper.

Primary Data

Creating primary data gives investigators intimate knowledge of their data and methods through the operationalization, collection, and analysis stages of a project. Simply, they know the data better than anybody, but in a methods section, they must attempt to communicate that knowledge to other researchers.

In articles based on primary data, we can identify four main sections: description of the unit of observation, design, measures, and procedure. Observation, the section describing the study, the universe of analysis, and the units, should include demographic information relating to the study. If, for example, individuals were surveyed, this could include the participant’s age, gender, ethnicity, year in school, or marital status. And if required, provide data as percentages, including descriptive statistics in mean and standard deviation. Design explains the type of study. Was it an experiment, survey, interview, or behavior observation? What type of observation was used? Why was this particular design chosen? Measures give information about the capture of observations and responses. Finally, procedure describes the process of the study exactly as it occurred.

Data Reuse

Articles using existing data do not need so much detail in explaining how the data were collected, as they can—hopefully—refer to the source of the data for information. However, researchers writing a paper reusing data have to convince readers of the suitability of these data and their usefulness in relation to the research question and hypotheses. This task becomes easier if the data are properly documented.

Finally, both types of papers need to properly describe the steps used to conduct their data analysis. The paper needs to explain why the methods employed are the most appropriate, considering the nature of the data and the precise formulation of the theoretical hypotheses. If applicable, the section should also describe the specific instruments of analysis used to study each research objective. Finally, it is now expected that authors provide a full, complete reference to the data as part of the reference list. This allows the original data collector to receive credit for the academic effort put into the data collection and allows researchers interested in replication, reuse, or education and training to find the data and allows for bibliometric analysis.

Expectations as to what should be in the methods section for empirical papers are, to a great extent, formalized. However, it was not always this way. The form and accuracy of the methods section depended mainly on the research practices adopted by the individual researcher or research team, and policies employed by the journal editor where the paper was published. Now, due to a significant focus on data sharing, replicability standards, and data management, this is no longer the case. In the next section, we look at how the data policies of research funders, expectations of RDM quality, and journal data policies have influenced methods sections and the replication movement.

Policies

The movement toward data sharing and openness has been facilitated by technology on one side, but also a political movement on the other. Cheaper, bigger and faster storage, transmission, and computational power—or the ability to do so—has been matched by the will to share data (see Anderson 2008). For example, in IR, the Correlates of War project brings together access to select quantitative IR datasets under the principles of “replication, data reliability, documentation, review, and the transparency of data collection procedures” (Correlates of War 2014). Big scientific challenges that can be addressed by technology and cooperation have brought data sharing agendas with them such as the Large Hadron Collider (LHC) ATLAS and LHCb and Human Genome Project 1990–2003 Bermuda Principles. This has coincided with declarations on open access (Budapest Open Access Initiative 2002, 2012) and open data in the form of the Berlin Declaration and organizational and governmental statements from the OECD (2007) and UNESCO (2012). The G8 Science Ministers Statement declared that publicly funded research data are a public good that should be made open and accessible to the fullest possible extent. The European Union (2010) and national research funders (IFDO Report 2014) have recognized that data-as-public-good arguments require data infrastructures to support data storage and sharing.

Yet, although the direction is clear, we have not yet arrived at the destination. There are still many research funders who do not require data sharing. IFDO class national funders can be grouped in three categories: those with data policies that include clear implementation procedures, those with an explicit data policy but without stated implementation suggestions, and a third group of funders without any policies. Of thirty-two nations, 52 percent do not have research funders with data policies, and of those that do, only Australia, Canada, the United States, the United Kingdom, Norway, Denmark, Sweden, and Finland have funders with clear recommendations on how to implement their data policies.

Data Management Quality

Since the 1960s, social science data archives have been active in managing and preserving research data. Yet there remains no universally accepted RDM standard. Each archive and data center adopted its own standards to fit the funder requirements and expectations of its user community. However, as research has become more cooperative and international, European social science data archives have actively sought to cooperate on standards of data description (DDI) and building a pan-European social science data infrastructure (CESSDA) supported by national governments. Although archives started in the beginning to offer basic archive and dissemination services, they advanced their data services to current needs offering advice and support on RDM and data sharing without necessarily ever archiving the research they were advising and supporting. But there has been an emergence of recognized international standards of digital preservation among archives (Data Seal of Approval 2014).

Data Policies of Academic Journals

Journal data policies requiring that the data and code underpinning articles be made available to readers have direct and indirect impact on the quality of methods sections. The direct impact is meeting the requirements of the policy in order to publish, requirements that often ask for an explanation of the process of data collection and analysis. The indirect impact on researchers creating primary data is that to comply with data availability requirements on availability, they have to ensure sufficient contextual information is provided to make the data comprehensible and document its collection to provide transparency. Data policies also challenge researchers to justify cases where data cannot be shared or shared with a limited audience rather than expect an exemption. In effect, these policies impose RDM requirements on researchers.

The adoption of data policies by journals is a big step toward the implementation of good replication standards. King (2003) proposed a checklist of what information should be included in a proper data availability policy that would ensure the replicability of empirical papers. This list includes not only the original data and code used to produce the results, but also the details on any specialized software, with an explanation of how to reproduce the exact output presented in a published article. These requirements provide confidence in the credibility of results (Freese 2007). Likewise, the PREPARDE project (2013) produced guideline requirements for data repositories to provide preservation and access to datasets as part of the scientific record. In 2014, a group of political science journal editors drafted a joint statement on Data Access and Research Transparency (DA-RT 2014, 2) that committed their journals to transparency, access, and availability and included a requirement that: “authors…delineate clearly the analytic procedures upon which their published claims rely, and where possible to provide access to all relevant analytic materials.”

A number of studies that have examined the presence and strength of data policies in political science (Gherghina and Katsanidou 2013), economics (Vlaemick and Siegert 2012), and sociology (Zenk-Moeltgen and Lepthien 2014) have contributed discipline-specific overviews. In IR, the pressure came from the researchers themselves. International Studies Perspectives organized a symposium on Replication in International Studies Research in which researchers shared their views on the importance of replication policies. The first step was a set of presentations at the 2002 annual meeting of the International Studies Association. The presenters then moved to a special issue of ISP with a set of articles that appeared in print in 2003. The result was a bottom-up imposition to adopt a single common replication policy by the four leading IR journals (Gleditsch, James, Ray and Russett 2003).

The journal’s impact factor is the single biggest influence for the adoption of data policies. Top-ranking journals are leaders in good research practice, and there is an interaction between journals adopting data policies and what is accepted as the norm of good scientific practice. Higher impact journals are more likely to have a data availability policy (Gherghina and Katsanidou 2013). However, applying a data policy gives a boost to the journal impact factor, too. IR journals that make their data available have a much higher impact than those that do not comply with this rule (Gleditsch, James, Ray and Russett 2003; Gleditsch, Metelis, and Strand 2003).

Implications for the Methods Sections

Based on this development, we expect to see that methods sections increasingly include elements essential to enable replication. The basic elements of a methods section are: a reference to data, data description, and variable description. These are essential components for replication and appear in almost all journal data policies in political science (Gherghina and Katsanidou 2013). Table 1 shows these elements. Each element is paired with specific indicators that need to appear in the methods section to ensure replicability. Given the growing call for and compliance with more rigorous data standards, we expect that more of the quality indicators in Table 1 will be present in empirical papers published closer to 2014.

Table 1

Elements of the methods section describing data

Elements of methods section Indicators 
Reference to data Referring to the data in the text 
 Indication where to find data 
 Full reference in the bibliography 
Data description Sample description 
 Reporting the response rate 
 Fieldwork description 
 Handling missing cases 
Variable description Question phrasing 
 Scales description 
Elements of methods section Indicators 
Reference to data Referring to the data in the text 
 Indication where to find data 
 Full reference in the bibliography 
Data description Sample description 
 Reporting the response rate 
 Fieldwork description 
 Handling missing cases 
Variable description Question phrasing 
 Scales description 

The distinction between papers using primary data and those reusing data remains an important one. We expect empirical papers using primary data will have higher standards in their methods due to the need to describe the data. Primary investigators of a study designed for sharing also use their publications to promote their data. Additionally, primary data are unknown and thus are expected to be presented in the paper. We expect that papers based on data reuse will reference the data description through citing the dataset so as to save space for the rest of essential methods section components.

Case Study: EVS

To investigate whether our expectations hold true, we use a case study. We have chosen the EVS, a survey that is widely used by Political Science, IR, and Sociology. The EVS is a large-scale, cross-national, and longitudinal survey focusing on how Europeans think about life, family, work, religion, politics, and society with repeat waves every nine years. The first EVS wave was 1981, when RDM was not well developed and there were no standardized documentation or metadata standards. Further waves took place in 1990, 1999, and 2008, with each wave surveying a larger sample in more European countries. In 1981, a thousand citizens in the EU member states of that time were interviewed. In 2008, the survey expanded to forty-seven European countries interviewing 70,000 people.

This study enables us to test our expectations because it stretches over time and space, but the conditions through which it was conducted have not changed much over time. First, it is an international study funded by a long list of different funders that includes funders at the forefront of the data sharing movement. Our expectation is that change in the research environment that came with the adoption of data policies will become clear. Second, every wave of the EVS had higher methodological standards and greater RDM needs. RDM throughout the whole lifecycle of these data is done by GESIS—Leibniz Institute for the Social Sciences (formerly the Zentralarchiv in Cologne). Since the 1980s, RDM at the GESIS data archive has become more professionalized, making the archive one of the leading survey data archives specializing in international comparative surveys. EVS experienced this increase in data management quality and became one of the important datasets in the GESIS archive. Thus, we expect to see the clear impact of increased RDM quality over time. Finally, EVS has been used as data source by its primary investigators and secondary users alike and has enabled them to produce empirical papers published in a wide range of journals. This enables us to test the impact of journal data policies on the methods sections of published papers using the same data source.

The empirical part of this paper is based on the coding of sixty-six published empirical articles from the period 1984–2013, all of which used datasets from all waves of the EVS. Table 2 shows how the sixty-six articles are divided among the four EVS waves. The articles were identified through the EVS Repository and Google Scholar using the search term “European Values Study.” We selected only empirical articles written in English and published in political science or sociology indexed in the Web of Science. Articles using multiple EVS waves are coded on the basis of the latest wave they have used to reflect the latest standards in methods sections. Other variables include whether the author belonged to the EVS group at the time of data collection, if the journal where the article was published had a data policy at the time of publication, as well as the indicators describing the three elements of high-quality methods sections from the side of data. All variables apart from the EVS waves were coded as dummies.

EVS waves Number of coded articles 
Wave 1: 1981 
Wave 2: 1990 11 
Wave 3: 1998 25 
Wave 4: 2008 20 
Sum 66 
EVS waves Number of coded articles 
Wave 1: 1981 
Wave 2: 1990 11 
Wave 3: 1998 25 
Wave 4: 2008 20 
Sum 66 

Descriptive Statistics

A first look at the descriptive evaluation of our data in Table 3 shows clear differences among the primary data and data reuse. Primary users make up 30.3 percent of our sample and data reusers 69.7 percent. Compared to data reusers, primary data users indicate more frequently where their data are to be found and provide full references in the biography, but they run slightly behind in references to the data in the text. Looking deeper, the articles were published after the first wave of EVS data collection when the survey was not fully consolidated and primary investigators also used the data for each country as independent data collections without referring to the study as EVS.

Table 3

Indicators of methods section quality for primary and secondary data users

 Primary data users (%) Data reuser (%) χ2 
In overall sample 30.3 69.7 – 
Referring to data source 90 100 4.74 
Indication where to find data 60 32.6 4.33 
Full reference in the bibliography 48 39 .67 
Sample description 80 67.4 1.08 
Reporting the response rate 35 23.9 3.57 
Fieldwork description 55 30.4 3.57 
Missing cases handling 55 65.2 .62 
Question phrasing 75 82.6 .51 
Scales description 80 73.9 .28 
 Primary data users (%) Data reuser (%) χ2 
In overall sample 30.3 69.7 – 
Referring to data source 90 100 4.74 
Indication where to find data 60 32.6 4.33 
Full reference in the bibliography 48 39 .67 
Sample description 80 67.4 1.08 
Reporting the response rate 35 23.9 3.57 
Fieldwork description 55 30.4 3.57 
Missing cases handling 55 65.2 .62 
Question phrasing 75 82.6 .51 
Scales description 80 73.9 .28 

Note: *p < .1

**

p < .05

Source: Self-collected data on journal articles using EVS data N = 66.

Data and variable description are also better documented by primary users than data reusers. Reporting the response rate and describing the field work proved to be statistically different between primary and secondary users. Despite the fact that data reusers have high scores in reporting data and variable description that enables users to have a better understanding of the data, they are definitely overtaken by primary data users. This might be due to the fact that data reusers restrict themselves to the information they consider relevant for the specific analysis at hand. Primary users, having themselves been involved in the collection and analysis, have a deeper knowledge of the data and want to give as much information as possible to the reader.

Table 4 shows descriptive statistics by EVS wave, dividing our coded papers according to the latest wave used starting with Wave 1 in 1981 through Wave 4 in 2008. The concentration of primary data users is higher in Waves 1 and 4 and lowest in Wave 3. The first element of the methods section referring to data is clearly better done in later waves. Referring to the data sources in a place within the body of the article starts at 88.9 percent in Wave 1 and reaches 100 percent in Waves 3 and 4. Indicating within the text where the data can be found and downloaded also increases over time. Giving a full reference of the data in the bibliography of the article is nonexistent in Wave 1, possibly as it was not common at that time, but reaches 42.9 percent by Wave 4.

Table 4

Indicators of methods section quality for the four EVS waves

 Wave 1: 1981 (%) Wave 2: 1990 (%) Wave 3: 1999 (%) Wave 4: 2008 (%) Total (%) 
Overall sample 13.4 16.4 38.8 31.3 100 
Primary data users 44.4 36.4 11.5 47.6 31.3 
Referring to data source 88.9 90.9 100 100 97 
Indication where to find data 11.1 36.4 42.3 57.1 41.8 
Full reference in the bibliography 36.4 42.3 42.9 35.8 
Sample description 88.9 36.4 65.4 90.5 71.6 
Reporting the response rate 22.2 9.1 26.9 38.1 26.9 
Fieldwork description 77.8 18.2 19.2 52.4 37.3 
Missing cases handling 13.4 10.4 37.3 13.4 74.6 
Question phrasing 55.6 63.6 92.3 80.9 79.1 
Scales description 77.8 45.4 84.6 80.9 76.1 
 Wave 1: 1981 (%) Wave 2: 1990 (%) Wave 3: 1999 (%) Wave 4: 2008 (%) Total (%) 
Overall sample 13.4 16.4 38.8 31.3 100 
Primary data users 44.4 36.4 11.5 47.6 31.3 
Referring to data source 88.9 90.9 100 100 97 
Indication where to find data 11.1 36.4 42.3 57.1 41.8 
Full reference in the bibliography 36.4 42.3 42.9 35.8 
Sample description 88.9 36.4 65.4 90.5 71.6 
Reporting the response rate 22.2 9.1 26.9 38.1 26.9 
Fieldwork description 77.8 18.2 19.2 52.4 37.3 
Missing cases handling 13.4 10.4 37.3 13.4 74.6 
Question phrasing 55.6 63.6 92.3 80.9 79.1 
Scales description 77.8 45.4 84.6 80.9 76.1 

Source: Self-collected data on journal articles using EVS data N = 66.

Overall, the descriptive results show that primary data users are more tentative about describing their data in the methods section. Time is also a significant component. The first wave seems to have had high-quality methods sections. We then see a drop in referring and describing the data, which changes in Waves 3 and 4, when replication standards become more relevant.

The second element of the methods section, data description, does not provide such clear results. In fact, we see a high point in Wave 1, then the frequency of reporting data description issues drops in Wave 2 only to increase again in Waves 3 and 4. This phenomenon could be explained by considering that in Wave 1, the EVS was innovative and those working with these data had an interest in describing them to promote the survey. Data description made their articles unique. The innovation wore off in Wave 2, but then data replication and RDM standards caught up, boosting data description in the methods section.

The third element of methods section is variable description. This shows a very different pattern. For question phrasing, we see an increase from Wave 1 to Wave 3 and a slight decrease in Wave 4. What we are seeing here might be the effect of referring to data documentation. Authors might not feel obliged to describe in the paper the working of questions from which their variables were derived, as they refer readers to documentation of the data. Scales description has a clear low in Wave 4 but remains quite high in the other time points.

Bivariate Statistics

In the next step, we investigate potential associations between the three elements of the methods section, describing data and three explanatory factors we described in the theoretical part of the paper: being a primary data user, time, and the journal having a data policy. We do so using bivariate correlations. Table 5 shows these associations.

Table 5

Correlations between indicators of methods section quality and influencing factors

 Primary data user Wave/time Data policy 
Referring to data source −.26 .24 .05 
Indication where to find data .27 .28 .34 
Full reference in the bibliography .03 .24 .14 
Sample description .14 .15 .05 
Reporting the response rate .1 .17 .21 
Fieldwork description .21 −.06 −.10 
Missing cases handling .02 −.34 −.09 
Question phrasing −.13 .23 .15 
Scales description .08 .14 .02 
 Primary data user Wave/time Data policy 
Referring to data source −.26 .24 .05 
Indication where to find data .27 .28 .34 
Full reference in the bibliography .03 .24 .14 
Sample description .14 .15 .05 
Reporting the response rate .1 .17 .21 
Fieldwork description .21 −.06 −.10 
Missing cases handling .02 −.34 −.09 
Question phrasing −.13 .23 .15 
Scales description .08 .14 .02 

Note: *p <.1

**

p < .05

***

p < .01

Bivariate correlation coefficients are reported.

Source: Self-collected data on journal articles using EVS data N = 66.

Being a primary data user has an impact only in the first methods component (referring to data). In fact, there is a negative association between referring to the data at all (in the text or in the bibliography) and being a primary data user. Interestingly enough, being a primary data user is also positively correlated with providing an indication in the text as to where to find the data.

Time is indicated here through a proxy: wave. What is hiding behind time is the increased quality of RDM and greater awareness of replication. Thus, we find time is the variable with the most significant associations. It has a positive impact on all three referring to data variables and reporting question phrasing. This indicates the data sharing and replication movement has some impact on the quality of methods sections. However, a contradictory finding is the negative impact on reporting the handling of missing cases.

Looking into more specific indicators of the data replication movement, we tested correlations with the existence of a journal data policy. Findings show a clear correlation between a policy and an indication of where to find the data used to produce the article. This is significant as the major component of journal data policies is expressed by the requirement of making the data used in the article available and accessible to the article’s readers. However, the rather low bivariate correlation of .34 shows that there is still a lot to be done to have a truly implemented data policy.

Multivariate Analysis

At this stage, we construct an index to incorporate all the above-mentioned elements of a good methods section. A factor analysis has shown that all items load on a single factor with eigenvalue 1.35. The new variable is an additive index scaling from zero to nine. We then use this additive index as the dependent variable in an Ordinary Least Squares (OLS) regression showing the causal impact of the three factors previously discussed. The independent variables used were tested for colinearity (tolerance 0.85) signifying that there is no need for caution.

In Table 6, we see being a primary data user and the latest wave of the data have a positive impact on the good quality of the methods section of the paper. The existence of a journal data policy has no significant impact. This result is the most puzzling. The suspected reason for this nonimpact is the lack of efficient enforcement. If a policy exists but remains a nicely written text in the journal website, then it is clear that authors take advantage and do not adhere to it. Thus, it is important for journal editors to make sure the policy is clear, make it known to the authors at the time of submission, and ensure that the authors have followed the policy before allowing the paper to be accepted for publication.

Table 6

OLS regression on the overall quality of a methods section

Constant 3.5 (.61) 
Primary data user .67 (.41) 
Wave/time .52 (.19) 
Data policy .49 (.42) 
N 66 
Adjusted R2 .107 
Constant 3.5 (.61) 
Primary data user .67 (.41) 
Wave/time .52 (.19) 
Data policy .49 (.42) 
N 66 
Adjusted R2 .107 

Note:*p <.1

**

p < .05

***

p < .01

This table reports b coefficients and standard errors are shown in parenthesis.

Source: Self-collected data on journal articles using EVS data.

Unfortunately, we were unable to test for the impact of the data policies of funders as only six studies in our sample referred to their source of funding, none of which referred to a funding agency that had produced a data policy at the time of article publication. These data would not have offered any insight, therefore, they were omitted from the analysis.

The main limitation of our results is that they cannot be generalized to a population of published articles using empirical data. What we analyzed is a convenience sample of published articles using one single comparative survey program. The results can be indicative of what happens in academic publishing in social sciences, but is by no means representative of the entire field.

Discussion

In a way, attempting RDM is a meta-phenomenon: research about research. Although the field of bibliometrics has developed this well, those working in RDM are still working out what it involves and approaches to implementation rather than studying the effect of its implementation. However, the hypotheses we were interested in testing focused on the effects of implementation. If funders and journals are pushing data sharing requirements, and if there is a scientific impulse toward replication, then we should be able to see the emergence of well-documented research to facilitate replication of data collection methods and/or analysis of data.

We have identified expectations as to how methods sections should be, and often are, structured. We controlled for papers using primary data or reusing data. We have outlined the general trend in data policies from funder to publication, actors, and the effect on good research practice as reflected in published articles. Finally, we empirically tested for differences over time from a general pre-RDM era to the current RDM conscious time using datasets from the four waves of the EVS between 1984 and 2013. This test clearly shows that general replication environment and RDM expectations have a positive impact on the quality of methods sections. What we also find is that journal data policies as they now stand do not improve replication standards.

The ideal of replicable behavioral political science is far from realized. Possible factors that may contribute to this are funders and journals that may have data policies but policies of varying strength and enforcement, so there is a degree of “wiggle room” between “should” and “must.” Likewise, if there is no enforcement and no checking up on failures to make data available, then efforts to promote replication standards are, ultimately, futile. There may also be a disincentive from the lower level of professional credit given to studies that attempt replication over original research. This is something that should change, as only with a culture of replication will researchers have a better appreciation of what information is required to replicate studies. Data infrastructures can only push standards so far, but if there is no will to accumulate experience of replication, then efforts at establishing standards of RDM will be in vain.

On a positive note, when a journal sets high standards, good-quality replicable research will follow. High-quality examples exist, coming from journals with a standing and enforced data policy such as International Studies Quarterly. In March 2013, the journal published four articles on the same topic, democratic peace, and debated the data and correctness of the empirical analysis used to reach different conclusions (Dafoe, Oneal, and Russett 2013; Gartzke and Weisiger 2013; Mousseau 2013; Ray 2013). This was a state-of-the-art research exchange that promotes scientific debate and ensures research quality.

1The data underpinning this article is available from Alexia Katsanidou, Laurence Horton, and Uwe Jensen (2014) “Data Policies, Data Management and the Quality of Academic Writing,” datorium. http://dx.doi.org/10.7802/70.
2American Political Science Review, American Journal of Political Science, Comparative Political Studies, Journal of Conflict Resolution, Journal of Theoretical Politics, Quarterly Journal of Political Science, Political Analysis, Political Science Research and Methods, State Politics and Policy Quarterly, and Research & Politics.
3For a list of EVS funders, see http://www.europeanvaluesstudy.eu/frmShowpage?v_page_id=2449946359342494.
4The EVS repository is found at http://www.europeanvaluesstudy.eu/evs/publications/.
5Description of variables in the Appendix.

References

Anderson
Chris
.
2008
.
“The End of Theory: The Data Deluge Makes the Scientific Method Obsolete,”
Wired Magazine
 
16
. Accessed February 4, 2015. http://www.wired.com/science/discoveries/magazine/16-07/pb_theory.
Berg
Bruce L.
1998
. “Writing Research Papers: Sorting the Noodles from the Soup.” In
Qualitative Research Methods for the Social Sciences
 .
3rd ed.
edited by
Berg
Bruce L.
,
253
72
.
Boston
:
Allyn & Bacon
.
Boyer
Mark A.
2003
.
“Abstract. Symposium on Replication in International Studies Research.”
International Studies Perspectives
 
4
:
72
107
.
Budapest Open Access Initiative
.
2002
. “Read the Budapest Open Access Initiative.” Accessed December 12, 2014. http://www.budapestopenaccessinitiative.org/read.
Budapest Open Access Initiative
2012
. “Ten Years on from the Budapest Open Access Initiative: Setting the Default to Open.” Accessed December 12, 2014. http://www.budapestopenaccessinitiative.org/boai-10-recommendations.
Bueno de Mesquita
Bruce
.
2003
.
“Getting Firm on Replication.”
International Studies Perspectives
 
4
:
72
107
.
Busenitz
Lowell W.
West
G. Page
Shepherd
Dean
Nelson
Teresa
Chandler
Gaylen N.
Zacharakis
Andrew
.
2003
.
“Entrepreneurship Research in Emergence: Past Trends and Future Directions.”
Journal of Management
 
29
:
285
308
.
Correlates of War
.
2014
. “Objectives.” Accessed August 6, 2014. http://www.correlatesofwar.org/objectives.htm.
Dafoe
Allan.
Oneal
John R.
Russett
Bruce
.
2013
.
“The Democratic Peace: Weighing the Evidence and Cautious Inference.”
International Studies Quarterly
 
57
:
201
14
.
Data Access and Research Transparency (DA-RT)
.
2014
. “A Joint Statement by Political Science Journal Editors.” Accessed December 14, 2014. http://media.wix.com/ugd/fa8393_da017d3fed824cf587932534c860ea25.pdf.
Data Seal of Approval
.
2014
. Accessed August 6, 2014. http://www.datasealofapproval.org/en/.
Easton
David
.
1953
.
The Political System: An Inquiry into the State of Political Science
 .
New York
:
Knopf
.
European Union
.
2010
. “Riding the Wave: How Europe Can Gain from the Rising Tide of Scientific Data. Final Report of the High Level Expert Group on Scientific Data.” Accessed February 4, 2014. http://cordis.europa.eu/fp7/ict/e-infrastructure/docs/hlg-sdi-report.pdf.
Freese
Jeremy.
2007
.
“Replication Standards in Quantitative Social Science—Why Not Sociology.”
Sociological Methods and Research
 
36
:
153
72
.
Gartzke
Erick
Weisiger
Alex
.
2013
.
“Permanent Friends? Dynamic Difference and the Democratic Peace.”
International Studies Quarterly
 
57
:
171
85
.
Gherghina
Sergiu
Katsanidou
Alexia
.
2013
.
“Data Availability in Political Science Journals.”
European Political Science
 
12
:
333
49
.
Gleditsch
Nils Petter
Metelits
Claire
Strand
Havard
.
2003
.
“Posting Your Data: Will You Be Scooped or Will You Be Famous?”
International Studies Perspectives
 
4
:
72
107
.
Gleditsch
Nils etter.
James
Patrick
Ray
James L.
Russett
Bruce
.
2003
.
“Editors’ Joint Statement: Minimum Replication Standards for International Relations Journals.”
International Studies Perspectives
 
4
:
105
.
IFDO Report
.
2014
. “Policies for Sharing Research Data in Social Sciences and Humanities.” Accessed August 6, 2014. http://ifdo.org/wordpress/wp-content/uploads/2014/04/ifdo_survey_report.pdf.
James
Patrick
.
2003
.
“Replication Policies and Practices at International Studies Quarterly.”
International Studies Perspectives
 
4
:
72
107
.
Katsanidou
Alexia
Horton
Laurence
Jensen
Uwe
.
2014
. “Data Policies, Data Management and the Quality of Academic Writing Datorium.” Accessed February 4, 2015. http://dx.doi.org/10.7802/70
King Gary
1995
.
“Replication, Replication.”
PS: Political Science & Politics
 
28
:
443
99
.
King Gary
2003
.
“The Future of Replication.”
International Studies Perspectives
 
4
:
72
107
.
King Gary
2006
.
“Publication, Publication.”
PS: Political Science & Politics
 
39
:
119
25
.
Mousseau
Michael
.
2013
.
“The Democratic Peace Unraveled: It’s the Economy.”
International Studies Quarterly
 
57
:
186
97
.
OECD
.
2007
. “OECD Principles and Guidelines for Access to Research Data from Public Funding.” Accessed August 6, 2014. http://www.oecd.org/sti/sci-tech/38500813.pdf.
Popper
Karl R.
1959
.
The Logic of Scientific Discovery
 .
London
:
Hutchinson
.
PREPARDE
.
2013
. “Guidelines on Recommending Data Repositories as Partners in Data Publication.” Accessed August 6, 2014. http://proj.badc.rl.ac.uk/preparde/attachment/wiki/DeliverablesList/D5_1_Generic%20Repository%20Accreditation%20guidelines_v8.pdf.
Ray
J. L
.
2013
.
“War on Democratic Peace.”
International Studies Quarterly
 
57
:
198
200
.
Ray
James Lee
Valeriano
Brandon
.
2003
.
“Barriers to Replication in Systematic Empirical Research on World Politics.”
International Studies Perspectives
 
4
:
72
107
.
Russett
Bruce.
2003
.
“The Journal of Conflict Resolution’s Policy on Replication.”
International Studies Perspectives
 
4
:
72
107
.
UNESCO
.
2012
. “Policy Guidelines for the Development and Promotion of Open Access.” Accessed August 6, 2014. http://unesdoc.unesco.org/images/0021/002158/215863e.pdf.
Vlaemick
Sven
Siegert
Olaf
.
2012
. “Welche Rolle spielen Forschungsdaten Eigentlich für Fachzeitschriften? Eine Analyse mit Fokus auf die Wirtschaftswissenschaften.” Working Paper 210, Working Paper Series des Rates für Sozial-und Wirtschaftsdaten. Berlin.
Zenk-Moeltgen
Wolfgang
Lepthien
Gesis
.
2014
.
“Data Sharing in Journals from Sociology.”
Online Information Review
 
38
:
709
22
. Accessed February 4, 2014. http://dx.doi.org/10.1108/OIR-05-2014-0119.

Appendix

Table A1

Descriptive statistics of variables used in the model as found in the dataset in Katsanidou, Horton and Jensen (2014)

Variable N Min. Max. Mean Standard deviation 
Latest wave used 66 2.88 1.01 
Author part of EVS team 66 .31 .47 
Journal having a data policy 66 .27 .45 
Reference to data      
Referring to the data in the text 66 .97 .17 
Indication where to find data 66 .42 .50 
Full reference in the bibliography 66 .36 .48 
Full reference in the bibliography with doi 66 .12 .45 
Data description      
Sample description 66 .72 .45 
Reporting the response rate 66 .27 .45 
Fieldwork description 66 0.37 .48 
Missing cases handling 66 .75 .44 
Number of cases 66 .78 .42 
Variable description      
Question phrasing 66 .79 .40 
Scales description 66 .76 .43 
Quality of methods section 66 5.04 1.79 
Variable N Min. Max. Mean Standard deviation 
Latest wave used 66 2.88 1.01 
Author part of EVS team 66 .31 .47 
Journal having a data policy 66 .27 .45 
Reference to data      
Referring to the data in the text 66 .97 .17 
Indication where to find data 66 .42 .50 
Full reference in the bibliography 66 .36 .48 
Full reference in the bibliography with doi 66 .12 .45 
Data description      
Sample description 66 .72 .45 
Reporting the response rate 66 .27 .45 
Fieldwork description 66 0.37 .48 
Missing cases handling 66 .75 .44 
Number of cases 66 .78 .42 
Variable description      
Question phrasing 66 .79 .40 
Scales description 66 .76 .43 
Quality of methods section 66 5.04 1.79