An Agenda for Open Science in Communication

In the last 10 years, many canonical findings in the social sciences appear unreliable. This so-called “replication crisis” has spurred calls for open science practices, which aim to increase the reproducibility, replicability, and generalizability of findings. Communication research is subject to many of the same challenges that have caused low replicability in other fields. As a result, we propose an agenda for adopting open science practices in Communication, which includes the following seven suggestions: (1) publish materials, data, and code; (2) preregister studies and submit registered reports; (3) conduct replications; (4) collaborate; (5) foster open science skills; (6) implement Transparency and Openness Promotion Guidelines; and (7) incentivize open science practices. Although in our agenda we focus mostly on quantitative research, we also reflect on open science practices relevant to qualitative research. We conclude by discussing potential objections and concerns associated with open science practices.

As Communication scholars, we aim to establish reliable and robust claims about communication processes.It is a bedrock of science that such claims are only reliable and robust if we can confirm them repeatedly.However, since 2010 several large-scale projects in various empirical sciences have shown that many canonical findings do not replicate (Camerer et al., 2016(Camerer et al., , 2018;;R. A. Klein et al., 2014R. A. Klein et al., , 2018;;Open Science Collaboration, 2015).The field of Communication has not yet conducted a large-scale replication project.However, because we employ similar methods to the fields that have already acknowledged these problems, there is reason to believe we face similar issues.The inability to replicate findings is troublesome for empirical disciplines, and it saps public trust in science (National Academy of Sciences, 2018).As a potential solution to this so-called replication crisis, a growing number of scholars have called for more transparent and open science practices (Nosek et al., 2015).Such practices tackle causes of low replicability and contribute to an ongoing credibility revolution (Vazire, 2019).Indeed, open science practices can increase replicability (Munafo `et al., 2017) and foster public trust (Pew Research Center, 2019).Therefore, in order to combat threats to the reliability and robustness of Communication scholarship, and to ensure that Communication research remains relevant in the public sphere, we believe that it is crucial for our field to act now and to implement open science practices.
To this end, we (a) discuss common causes of low replicability in science broadly, before we (b) outline growing concerns about a replication crisis in Communication particularly.We (c) explain how open science practices can address these concerns by offering an agenda of seven specific solutions for implementing open science practices in Communication.Although we focus mostly on confirmatory and quantitative research, we also (d) reflect on open science practices relevant to other approaches such as qualitative research.Finally, we (e) discuss potential concerns and objections against the proposed open science practices.
Whereas we are not the first to identify these problems and solutions, our main contribution is that we build on the insights generated by other adjacent disciplines and apply these to Communication.We define the most important problems in our field, identify potential solutions, and provide a concrete plan of action.Ultimately, we believe that by following these solutions, we as Communication scholars can collectively improve and update the empirical basis for our understanding of how communication processes unfold in the many contexts we study.

Causes of low replicability
Before we outline the causes of low replicability, we briefly define the underlying concepts.Replicability means that a finding "can be obtained with other random samples drawn from a multidimensional space that captures the most important facets of the research design."(Asendorpf et al., 2013, p. 109).Reproducibility means that "Researcher B [. ..] obtains exactly the same results (e.g., statistics and parameter estimates) that were originally reported by Researcher A [. ..] from A's data when following the same [data analysis]" (Asendorpf et al., 2013, p. 109).Importantly, for most quantitative researchers the end goal is generalizability, which means that a finding "[. ..] does not depend on an originally unmeasured variable that has a systematic effect.[. ..]Generalizability requires replicability but extends the conditions to which the effect applies" (Asendorpf et al., 2013, p. 110).
Several large-scale projects have examined the replicability of scientific findings in various fields such as psychology and economics (for an overview, see Online Appendix SB).Replication rates in these projects vary considerably, ranging from 36% in cognitive and social psychology (Open Science Collaboration, 2015) to 78% in experimental philosophy (Cova et al., 2018).Importantly, there is no consensus on what constitutes an appropriate replicability rate or even measure.Nevertheless, these projects show that a substantial amount of findings in neighboring disciplines do not replicate.Whereas a systematic investigation of replicability in Communication is lacking, it is plausible that we face similar issues given the overlap in research methods and publishing practices.
Building on Bishop (2019), we identify four major causes of low replicability.On the side of the researcher, a substantial challenge to robust science are so-called questionable research practices.On the side of journals-which includes editors, board members, and reviewers-a preference for novel and statistically significant results creates a publication bias.In the social sciences, many fields investigate small effects whilst relying on small samples, which leads to low statistical power.Last, replicability is reduced by problems resulting from human errors, which include the false reporting of statistical results.

Questionable research practices
Quantitative Communication scholars usually rely on empirical data collected, for example, via questionnaires, observation, or content analyses.In order to determine the generalizability of the results, a common approach to analyzing these empirical data is the use of frequentist null hypothesis significance testing (NHST; Levine, Weber, Hullett, Park, & Lindsey, 2008).In NHST, we calculate the probability of the empirical data (or more extreme data) under the null hypothesis.If the probability of the empirical data (or more extreme data) is below a specific threshold, we consider our results statistically significant, which leads us to reject the null hypothesis.In the social sciences, including Communication, we have settled for the (arbitrary) threshold of a ¼ 5%.
The adoption of this threshold has led to the false belief that statistical significance represents a benchmark of "real" effects and/or high-quality research.Unfortunately, this encourages researchers, often unknowingly, to engage in so-called "questionable research practices" (QRPs; John, Loewenstein, & Prelec, 2012, p. 524).QRPs are aimed solely at achieving statistical significance (i.e., p-values less than 5%).QRPs are both widespread and difficult to recognize (John et al., 2012).Critically, they do not even require ill intent on the side of the researcher (Gelman & Loken, 2013).In fact, many QRPs have been considered standard procedure and are part of many training programs (Wagenmakers, Wetzels, Borsboom, & van der Maas, 2011).Below, we explain two prominent QRPs, HARKing and p-hacking, in more detail.

HARKing
Knowledge generation typically relies on two distinct modes of research.In exploratory research, new hypotheses are generated; in confirmatory research, a priori formulated hypotheses are tested.Both modes of research serve different functions and are crucial for scientific progress.A confirmatory approach is most relevant to the self-corrective nature of science.From a falsification paradigm, we are compelled to abandon predictions that do not reliably obtain empirical support (Popper, 1959), which helps discard unfruitful research avenues.Conversely, an exploratory approach can be used to articulate postdictions, which can help develop or update theories.
A substantial problem arises when researchers present exploratory research as if it were confirmatory research; that is, when they label postdictions as predictions.This QRP is known as HARKing, an acronym for Hypothesizing After Results are Known (Kerr, 1998).When HARKing, data are used to generate hypotheses that are tested on the same data (Nosek, Ebersole, DeHaven, & Mellor, 2018).To illustrate, imagine a researcher who expects Condition A to be more effective than Condition B. However, when results reveal that Condition B is more effective, the researcher writes the manuscript as if they had expected Condition B to be more effective all along.Therefore, HARKing constitutes circular reasoning.It fails the very purpose of hypothesis testing, and it violates the basic scientific method.Crucially, HARKing capitalizes on chance: Unexpected results might not represent stable effects, which dilutes the literature with false positives and contributes to low replicability (Nosek et al., 2018).

p-Hacking
When analyzing data, there are multiple legitimate analytic options, so-called "researcher degrees of freedom" (Simmons, Nelson, & Simonsohn, 2011, p. 1359).As a result, researchers find themselves in the proverbial "garden of forking paths" (Gelman & Loken, 2013, p. 1).Some paths will lead to statistically significant results, others will not.The situation becomes particularly problematic when researchers deliberately search and choose those paths that lead to significance, a practice known as p-hacking (Simmons et al., 2011).
For example, when conducting a multiple regression analysis that does not reveal a significant result, researchers might include (or remove) statistical control variables, which increases their chances of obtaining a statistically significant result.Other examples of p-hacking include (a) continuing data collection until researchers find significant results, (b) using multiple measures of a construct and reporting only those with statistically significant results, (c) including or excluding scale items depending on whether or not they produce significance, (d) including or excluding outliers from data analysis to achieve significance, and (e) selecting and analyzing only specific subgroups that show a significant effect.
Using these QRPs, Simmons et al. (2011) demonstrated "how unacceptably easy it is to accumulate (and report) statistically significant evidence for a false hypothesis" (p.1359).They showed that p-hacking increases the chances of finding statistically significant results for non-existent effects by as much as 60%. 1 As with HARKing, p-hacking results in effects that are neither reliable nor robust, which clutters the literature with non-replicable findings.

Publication bias
Statistically significant findings are more likely to get published than non-significant ones, which creates a so-called publication bias (Ioannidis, Munafo `, Fusar-Poli, Nosek, & David, 2014).Scholars can contribute to this bias as authors, reviewers, and editors.
First, many authors feel that nonsignificant findings do not contribute anything substantial to the literature.As a result, nonsignificant results often remain unpublished, creating the so-called "file-drawer problem" (Rosenthal, 1979, p. 638).To illustrate, Cooper, DeNeve, and Charlton (1997) surveyed a small sample of social scientists to ask about the fate of studies approved by their institutional research board.They found that statistically significant findings were far more likely to be submitted to peer-review than non-significant findings.Second, reviewers and editors often reject manuscripts because they consider the results not sufficiently novel, conclusive, or exciting (Giner-Sorolla, 2012)-a tendency especially apparent in the case of failed replications (Arceneaux, Bakker, Gothreau, & Schumacher, 2019).As a consequence, authors are encouraged either to discard studies in which some predictions are supported but others are not or, even worse, to actively engage in p-hacking to achieve a coherent story and a definitive conclusion (O'Boyle, Banks, & Gonzalez-Mule ´, 2017).
In conclusion, although the extent to which manuscripts with null findings are rejected is not known, publication bias leads to an overrepresentation of both significant findings and inflated effect sizes (Fanelli, 2012).These practices create a bizarre situation in which effects that appear well-supported in the literature actually do not exist, resulting in the canonization of false facts (Nissen, Magidson, Gross, & Bergstrom, 2016) and, ultimately, low replicability.

Low statistical power
Power refers to the probability of observing a true effect.For typical between-person designs, power is determined by the alpha level, the true effect size and variance in the population, the sample size, study design, and the type of hypothesis or statistical test (e.g., one-versus two-sided; Cohen, 1992).Broadly, for large effects, small samples can reliably detect effects; for small effects, large samples are needed (Cohen, 1992).In practice, researchers can determine an adequate number of cases for a specific effect by conducting a priori power analyses (e.g., using tools such as G*Power or the R package pwr).When researchers analyze a small effect with a small sample, analyses are underpowered.Underpowered analyses are highly problematic: First, they reduce our ability to find effects that actually exist.Second, they overestimate the size of those effects that are found (Funder & Ozer, 2019).Thus, low power leads to erroneous results that are unlikely to replicate.

Human errors
All humans make errors; all researchers are humans; hence, all researchers make errors.An analysis of more than 250,000 psychology papers published between 1985 and 2013 found that half of those reporting significance tests contained at least one p-value inconsistent with its test statistic or degrees of freedom (Nuijten, Hartgerink, van Assen, Epskamp, & Wicherts, 2016).Although many of these errors are unintentional, researchers seem reluctant to share their data in order to help detect and correct errors.For example, Vanpaemel, Vermorgen, Deriemaecker, and Storms (2015) found that less than 40% of the authors who published a manuscript in one of four American Psychological Association (APA) journals in 2012 shared their data upon request, even though a refusal to share is a violation of APA research ethics (American Psychological Association, 2009, p. 12).Even when statistical reporting errors are detected in published research, issuing corrections is arduous (e.g., Retraction Watch, 2018).Human errors are a natural by-product of science as a human enterprise and need to be expected, but the current system is not designed to detect, embrace, or correct mistakes.As a result, the literature contains too many erroneous findings, which is another reason for low replicability.

Replicability in Communication
Given the obvious overlap of methods, theories, and publication practices in the quantitative social sciences (Zhu & Fu, 2019), we have reason to believe that Communication also suffers from low replicability.Indeed, there are early warning signs that our discipline has a replication problem.A special issue of Communication Studies reported nine replication attempts of published Communication studies (McEwan, Carpenter, & Westerman, 2018).The results resemble those of prior replications projects: Two studies replicated all of the prior findings, three studies replicated some of the prior findings, whereas four studies replicated none of the prior findings (McEwan et al., 2018).Together, these results suggest low replicability.This is not surprising given that the same four causes of low replicability and reproducibility can also be found in Communication.For example, studies in leading journals in Communication show evidence of the QRPs discussed above (Matthes et al., 2015), demonstrating that we engage in the same practices as our colleagues in other fields.Likewise, there is a growing body of studies in Communication illustrating the "garden of forking paths," showing that analytical results can differ starkly depending on the analytical choices made by the researchers (e.g., Orben, Dienlin, & Przybylski, 2019).Similarly, there exist several accounts of publication bias in Communication (Franco, Malhotra, & Simonovits, 2014;Keating & Totzkay, 2019), which indicates a preference for novel and statistically significant results.Next, just like all other scientists, Communication scholars commit errors: Of all p-values reported in 693 empirical Communication papers published between 2010 and 2012, 8.8% were incorrect; of those, 75% were too low (Vermeulen & Hartmann, 2015).Similarly, when it comes to correctly interpreting p-values, Communication scholars commit errors (Rinke & Schneider, 2018).
A strong indicator of low replicability is low statistical power.In their meta-review of 60 years of quantitative research, Rains, Levine, and Weber (2018) reported that observed effects in Communication have a median size of r ¼ .18.In other words, using traditional benchmarks (Cohen, 1992) effects are typically small to moderate (but see Funder & Ozer, 2019).However, note that this is an overly optimistic assumption in light of QRPs and publication bias; not all effects that are reported actually exist, and those that do are likely smaller (Camerer et al., 2018).Next, Rains et al. (2018) report that meta-analyses in Communication typically feature a median of 28 effects and a median of 4,663 participants, from which we can extrapolate that effects are typically tested with 167 participants.However, for most study designs such a small sample size would be insufficient to reliably detect small or moderate effects (for examples of specific designs, see Online Appendix SC)-which suggests that a large number of studies in Communication are likely to be underpowered.
The presence of these four causes of low replicability implies that we have a replication problem in Communication as well.Similar to other fields, in Communication "there is an increased internal understanding of our own faults and foibles that is leading to requests for more information about what underlies the evidence that serves as a basis for our knowledge claims" (Holbert, 2019, p. 237).Hence, we must not wait for a large-scale replication project to further demonstrate that substantial portions of our research cannot be replicated.Instead, we believe that we must act now.Open science practices can be an important part of ensuring our research is reproducible, replicable, and thus credible (Munafo `et al., 2017).

An agenda for open science practices in Communication
Open science is an umbrella term and describes "the process of making the content and process of producing evidence and claims transparent and accessible to others" (Munafo `et al., 2017, p. 5).In other words, open science practices shift the research focus from a closed "trust me, I'm a scientist" to a more transparent "here, let me show you" position (Bowman & Keene, 2018, p. 364).In what follows, we present the seven solutions that we consider most relevant for addressing low replicability (see Table 1).Whereas our first points focus on direct solutions, the actual research, and the individual researcher (see also Lewis, 2019), our final points emphasize more indirect approaches and also include other stakeholders.

Publish materials, data, and code
We recommend that researchers share study materials (e.g., items, stimuli, protocols, instructions, or codebooks for content-and meta-analyses), data (e.g., raw, aggregated, processed, or synthetic), code (e.g., data-gathering, -preparation, or -analysis), and software (e.g., operationalizations of experiments, simulations, content-coding tools, scraping tools, or applications) when appropriate and ethical.These recommendations are aligned with the International Communication Association's (ICA) Code of Ethics, 2 which states that "ICA fully supports the openness of scholarly research" (p.2).When sharing, we recommend following the FAIR data principles (an acronym for findability, accessibility, interoperability, and reusability; Wilkinson et al., 2016) or the suggestions offered by O. Klein et al. (2018).
Sharing research materials has several important benefits.First, sharing of materials, data, code, and software increases reproducibility, because others can independently verify and better understand the results (LeBel, McCarthy, Earp, Elson, & Vanpaemel, 2018).As an immediate result, sharing improves the quality of peer review.Whereas some peer-review criteria can be assessed on the basis of the final manuscript (e.g., the strength of the theoretical rationale), other criteria require access to (a) study materials (to comprehensively assess a proposed methodology), (b) analysis code (to better evaluate a data analysis), or (c) study data (to check results for potential errors).Without access to these materials, reviewers must accept scientific claims solely based on authors' claims (Munafo `et al., 2017).Second, sharing improves the quality of the research itself, because it provides the basis for cumulative knowledge generation.Science is ultimately a social enterprise in which individual researchers and groups work together to create and to accumulate knowledge as a public good (Merton, 1974).In each research project, scholars rely on prior work to develop theories, craft research designs, or develop analytical procedures.Importantly, the quality of each step depends on how much information about prior work is available.Sharing allows researchers to work more efficiently and conceive new studies without having to "reinvent the wheel." 2. Preregister studies and submit registered reports Many QRPs are not a result of bad intentions.Like all humans, researchers are inclined to see patterns in random data that are aligned with preexisting beliefs (Munafo `et al., 2017).Therefore, introducing structures that limit biases as part of the scientific process is beneficial.To this end, we recommend that all confirmatory research should be preregistered.(Much exploratory research can be preregistered, too.)Preregistration means that hypotheses, study design, and analysis plan are explicated in an official registry prior to data collection or data analysis (Nosek et al., 2018).Preregistration platforms such as AsPredicted 3 and OSF Registries 4 also ask for a justification of the planned sample size.To justify their sample and to prevent low power, we recommend that researchers conduct a priori power analyses when planning a study.Preregistrations can also include additional details, such as a short summary of the project, measures or coding schemes, variable transformations, recruitment or sampling strategies, and power calculations.This initial research plan is preserved in the registry, receives a time-stamp, is made discoverable (if desired, only after an embargo period), and is linked to in the research article (so that planned and conducted analyses can be compared).For further instructions and concrete templates, see Lewis (2019).
Because preregistration means all steps are determined before data collection, researchers protect themselves from HARKing and p-hacking, which reduces the likelihood of false or inflated effects.Some evidence of this effect already exists-for example, a comparison of the effect sizes from 993 studies in psychology found that preregistered studies reported effects (r ¼ .16)that were less than half as large compared to those reported by studies that were not preregistered (r ¼ .36;Scha ¨fer & Schwarz, 2019).Although this result could be a selection effect, it has been demonstrated elsewhere (e.g., Camerer et al., 2018).Critically, preregistration does not prevent authors from post-hoc exploratory data analyses, but only requires authors to clearly distinguish confirmatory and exploratory analyses.Exploratory analyses are crucial for scientific advancement; they should not be disguised as confirmatory analyses but receive a designated section and thereby a more prominent spot.
A logical extension of preregistrations are registered reports (Chambers, 2013).Registered reports follow a two-stage review process.In Stage 1, authors submit a manuscript that includes the introduction, method, planned sample size, and any pilot study results.This proposal is sent through the usual peer-review process before the study is conducted.Reviewers assess the merits of the study design.If evaluated positively, authors receive an in-principle acceptance, which-as long as the research is conducted as specified in the accepted proposal-guarantees publication of the manuscript, regardless of results.In Stage 2, the authors submit the full manuscript, which includes the results of their preregistered analysis plan, exploratory results, and a discussion of their findings.Deviations from the preregistration have to be highlighted and explained.In the second round of peer review, reviewers evaluate if the confirmatory analyses correspond to the planned procedure, assess any new exploratory analyses, and give feedback on the discussion.
Registered reports provide peer review when it has the most impact: before the study is conducted.Hence, in contrast to traditional submission types, registered reports can improve the design of the research.Moreover, given that publication is independent of a study's outcome, registered reports eliminate publication bias (Munafo èt al., 2017).Several journals have offered exploratory reports as a format dedicated to hypothesis generation and discovery (McIntosh, 2017).The list of journals offering registered reports is growing continually, already listing more than 200 journals (Chambers, 2019).In Communication, at the time of writing, registered reports are accepted by Communication Research Reports, Computational Communication Research, and Journal of Media Psychology.We urge other Communication journals to follow suit and encourage scholars to submit their work as registered reports.

Conduct replications
We encourage (a) Communication researchers to conduct more replication studies and (b) editors and reviewers to publish more replication studies.Although conducting and publishing replications is important for scientific progress in general (Merton, 1974), it is central to open science in particular, because it makes transparent the robustness of previously published results.
At least three types of replications exist (for a more granular conceptualization, see LeBel et al., 2018): direct replications, when a researcher reruns a study using the same operationalizations and data analysis methods; close replications, when a researcher for example updates the stimulus material (Brandt et al., 2014); and conceptual replications, when a researcher reruns parts of a study or uses different operationalizations/methods.Notably, internal replications, in which researchers replicate their own work, are not predictive of independent replication efforts (Kunert, 2016).Trying to replicate the results of other researchers is therefore a requirement of the scientific method and a necessity for science to be self-correcting.
Although conceptual replications of research are not uncommon, there is a shortage of direct replications.In Communication, they represent approximately 1.8% of all published research (Keating & Totzkay, 2019).Thus, we call upon both editors and reviewers to be open to the publication of direct replications.Specifically, we believe that journals have a responsibility to publish high-quality replications of studies that the journal originally published; a procedure known as the "Pottery Barn rule" (Srivastava, 2012).This challenges journals to find ways to support the submission of high-quality replications.Authors struggling to find venues for the publication of replications can self-publish their research as preprints (Berg et al., 2016).
There are several guidelines for conducting replications (e.g., Brandt et al., 2014).Here, we emphasize three aspects.First, replications should not be used as a political tool to disparage individual researchers.Instead, they inform our research and update our knowledge about important effects.Second, although not necessarily a condition for a good replication, we encourage researchers to contact the authors of the original study to get feedback on their preregistered replication attempt.Third, it is a common misunderstanding that direct replications should rerun the old studies using the same sample size.Given publication bias, small samples, and inflated effect sizes, replications need new power analyses, preferably assuming that the actual effects are half as strong as initially reported (Camerer et al., 2018).It is crucial that replication attempts have sufficient statistical power, especially in cases when the original study was underpowered.For well-executed replication attempts, see Camerer et al. (2018).

Collaborate
Because effects in Communication are typically small to moderate (Rains et al., 2018), a priori power analyses often reveal that we need large samples to reliably detect such effects.However, large samples necessitate vast resources.As a result, we encourage Communication scholars to collaborate across labs or research sitessomething routinely done in other fields (e.g., the Human Connectome project 5 ).Such collaborations can take place on a small scale (with a few individuals or labs joining forces), on a large scale (with dozens of labs participating worldwide), or indirectly by analyzing already existing large-scale datasets or by cooperating with companies.More collaboration reflects the basic idea of open science in the sense of strengthening interactions among scholars, enabling a more proactive exchange of data and study materials, and establishing the mindset that we need large scale empirical data to produce reliable results, which can be collected and maintained only by a collective effort.
In order to find collaborators for small-scale collaborations, researchers can use online resources such as StudySwap, a platform for interlab replication, collaboration, and research resource exchange. 6Regarding large-scale collaborations, programs such as the Many Labs projects involve several labs working together in order to replicate contemporary findings (e.g., R. A. Klein et al., 2018); for an example of a large-scale cooperation in Communication, see Hameleers et al. (2018).Next, Communication scholars might consider using existing large-scale datasets.For a list of freely available datasets, see Brick (2019); for a search engine for datasets, see Google Dataset Search 7 or GESIS Data Search. 8Collaborations with companies such as Google or Facebook 9 are another option (Taneja, 2016).

Foster open science skills
In order for open science practices to become widely adopted, it is essential that they become an integral component of a researcher's training and education (van den Berg et al., 2017).Good education and training are the cornerstones of highquality research; hence, they can treat the causes of low replicability at their roots.
Broadly, researchers can make use of open access learning resources focused on open science practices.These services include webinars, teaching materials, or consultation, offered by the Center for Open Science, 10 FOSTER Plus, 11 and many other organizations.Furthermore, there exist several massive open online courses (so-called MOOCs), 12 video material, 13 an Open Science Knowledge Base, 14 or tutorials (e.g., O. Klein et al., 2018), which all help develop a familiarity with or expertise in open science practices.
Second, researchers should encourage students to implement open science practices as part of the advising and mentoring process.For example, students could (a) preregister theses and studies conducted for their coursework, (b) conduct replication studies, which is ideal for understanding important methods and theories in Communication while simultaneously contributing valuable epistemic insights, or (c) analyze publicly available datasets, which significantly expedites the research process while being able to produce reliable findings (Scho ¨nbrodt, Maier, Heene, & Zehetleitner, 2015).Fortunately, it is often possible to build on established practices and routines: Similar to preregistrations, thesis projects often require students to first propose their theoretical foundations, study design and methodology, and planned data analysis (Nosek et al., 2018).Likewise, students are often required to share data and analysis scripts with their advisor and other advisory committee members, which could easily be extended to open data repositories.

Implement Transparency and Openness Promotion Guidelines
Replicability can be increased indirectly by promoting an open research publication culture.This can be achieved if academic journals adopt, promote, and require open science practices.To structure this process, Nosek et al. (2015) proposed the so-called Transparency and Openness Promotion (TOP) Guidelines, which consist of eight standards that largely encompass the suggestions outlined in the current manuscript.Broadly, the TOP Guidelines encourage journals to ensure that as much of the work published in their outlets is made available to the public, while clearly communicating how authors and readers should engage those materials.
TOP Guidelines acknowledge that not all open science practices are possible or plausible for all areas of research.They propose three incremental levels of transparency and openness.Level 1 necessitates only an update of submission guidelines, in which submitting authors are required to state whether they shared their data, code, or materials; actual sharing is encouraged but not required.On Level 2, journals require authors to share their data, code, and materials on trusted repositories.
Finally, Level 3 represents a move toward complete transparency, in which journals adopt all open science practices suggested by TOP.This includes the preregistration of all confirmatory studies and the enforcement of design and analysis transparency standards.As of this writing, TOP Guidelines are adopted by over 1,000 journals. 15 Within Communication, there are explicit calls for more transparency and open sharing (Bowman & Keene, 2018;Lewis, 2019;Spence, 2019).We therefore encourage Communication journals to adopt the TOP Guidelines.This change needs joint efforts from various stakeholders, including publishers, publication committees, editors, board members, reviewers, and authors.

Incentivize open science practices
Only by changing academia's incentive system will it be possible to guarantee sustained change.An implicit incentive that already exists may be the reputational gain that can be achieved through the early adoption of open science standards (Allen & Mehler, 2019;McKiernan et al., 2016).Furthermore, the opportunity to publish null findings via registered reports may also contribute to traditional markers of productivity such as number of publications.However, to combat low replicability and its causes effectively, Communication needs to incentivize open science practices explicitly.
Above all, it is crucial that we introduce the successful implementation of the abovementioned practices of open science as a quality indicator to selection and evaluation processes, which includes hiring, tenure, promotion, and awards.To this end, several universities have already begun to require that applicants list practices of open science (Allen & Mehler, 2019).Funders, including the European Commission 16 and the German 17 and Dutch 18 Science Foundations, increasingly require explicit open science practices.On the side of journals, introducing badges that signal a manuscript's adherence to open science practices (e.g., open material, open code, open data) may incentivize open science practices (Kidwell et al., 2016).
Changing our incentive structure necessitates changing our general culture.This can only be achieved if we come together as a community, and events such as the 2020 ICA conference "Open Communication" express an urge to make a change.In that vein, preconferences, theme slots, or symposia are great ways to further engage the community.Local and decentralized grassroots initiatives (e.g., the open science journal club "ReproducibiliTea" by Orben (2019) or the "UK Reproducibility Network") 19 might similarly affect sustained cultural changes.

Open science and qualitative research
To this point, we have implicitly focused on quantitative research.Open science practices, however, are not exclusive to any particular form of data or type of analysis.The basic notion of making scholarship transparent is one shared by all scholars (Haven & Van Grootel, 2019).That said, reasons and motivations for engaging in open science differ across approaches, as do implementations and solutions.It is beyond the scope of this paper to address all aspects that need to be considered for the manifold approaches that are used within Communication.Instead, we want to emphasize that other approaches can similarly benefit from engaging in open science practices.In what follows, we briefly outline some first suggestions as to how open science practices can be used to strengthen qualitative research.
Because there are different plausible understandings of what constitutes qualitative research, we refer here to approaches that (a) aim at understanding how and why certain phenomena may occur, instead of making inferences about the larger population from which the sample was drawn; and (b) involve complex data (e.g., texts, videos, images) that are analyzed using semantic approaches (e.g., verbal interpretation, categorization, encoding).Data often come from smaller samples that do not maximize representativeness but rather heterogeneity, the interpretation is carried out by the researchers themselves, and if statistics are reported at all (e.g., frequencies within the small sample), they are primarily descriptive.Due to their comprehensive and more granular nature, findings from such methods contribute to a better understanding of specific processes and offer new insights that are less likely to be produced or even impossible to produce with quantitative methods.
Because qualitative research is primarily based on subjective evaluations and does not include inferential statistics, many of the abovementioned QRPs such as p-hacking or low statistical power do not apply.For the same reason, qualitative research cannot really have a replication crisis, because it is not the explicit aim to generalize over an underlying population (although mixed method approaches do exist, which are at least partly confirmatory).Similarly, because an individual human researcher is centrally involved in the interpretation of the data, general reproducibility is by definition limited (Childs, McLeod, Lomas, & Cook, 2014).That said, qualitative research is not entirely subjective.Its methods are rooted in general principles shared by many researchers, which allows us to compare and evaluate results (e.g., when determining interrater reliability).Because qualitative research informs us about underlying processes it can establish a profound understanding of specific mechanisms, particularly about those that a researcher has not thought of a priori.Together, this still implies a certain but much more modest aim when it comes to generalizing results.Again, acknowledging that different understandings, practices, and aims exist, it is our understanding that several of the abovementioned open science practices will also benefit qualitative research.In short, we argue that open science practices can increase the quality of qualitative research primarily by improving transparency and traceability.
First, researchers can share research designs, interview and interrogation protocols, anonymized data and coded data files, and coding strategies used to analyze these data (Haven & Van Grootel, 2019).Haven and Van Grootel (2019) assert that working with qualitative data does not "exempt the researcher from the duty to maximize transparency" (p.236).This approach allows other scholars to better understand their interpretive lenses, to better assess the quality of the findings, and to use or adapt these materials in their own research (DeWalt & DeWalt, 2011).Together, this increases consistency and comparability.
Scholars working with qualitative analyses have many researcher degrees of freedom as well, and are incentivized to find compelling narratives that increase chances of publication.Together, this introduces biases that can reduce precision and, in turn, generalizability.Hence, preregistration of qualitative research can be fruitful, too: It increases transparency, tracks flexibility and modification during the research process, and, when submitted as a registered report, prevents publication bias (Haven & Van Grootel, 2019).For the most part, preregistering a qualitative study is similar to a quantitative study.It includes specifying the a priori choices for sampling frames, data collection tools, or planned data analysis choices.In contrast to quantitative research, preregistrations need not be about registering predictions, but "putting the study design and plan on an open platform for the (scientific) community to scrutinize" (Haven and Van Grootel, 2019, p. 236).Such a process would motivate researchers to "make explicit which tradition and theoretical lens they work from" and to "carefully reflect upon their own values prior to going into the field and prior to interpreting and reporting the findings within the context of these a priori values" (Haven and Van Grootel, 2019, p. 237).Preregistration can thereby provide an additional layer of accountability and credibility for the work as a whole.For a preregistration template for qualitative research, see Hartman, Kern, and Mellor (2018).
We encourage scholars conducting qualitative research to collaborate as well, for example through triangulation (Creswell & Poth, 2018), secondary data analysis, or by using multiple datasets of qualitative phenomena (Ruggiano & Perry, 2019).Each of these possibilities, or any combination of them, allows for deeper and wider insights into the research questions at hand and would help examine the relative stability, variability, and generalisability of a given result.If a number of researchers with different interpretive lenses find similar or complementary results, we can be more confident in the claims of this collective research.

Objections and concerns
Some might question whether or not Communication really has a replication problem.Throughout this manuscript, we have presented several arguments as to why there is cause for concern (and for a list of further reasons, see Online Appendix SD).However, even if our analyses should be incorrect, consider that we as Communication scholars have always aimed to improve our scholarly practices over time.Because our agenda is built on shared principles of science (Merton, 1974), we believe that adopting open science practices represents a natural continuation of this collective development that will make our research more informed, robust, and credible.Some might express concerns over specific open science practices.One of the most prominent concerns regards data sharing.Sometimes the sharing of materials or data is problematic or unethical, because they could be used for unintended, harmful, or unethical purposes (Lakom y, Hlavova ´, & Machackova, 2019).One key issue is the privacy of participants.There are cases in which the full data cannot be shared because participants can or could be identified.Specific raw data such as news articles, video material, or data scraped from online platforms sometimes cannot be made public, for example because of copyright reasons (van Atteveldt, Strycharz, Trilling, & Welbers, 2019).Fortunately, there are several means to address these challenges.First, it is necessary to implement an appropriate informed consent process, in which participants are informed about who has access to the data and how they can delete their data.Second, researchers can prevent others from identifying participants by (a) removing direct or indirect identifiers (e.g., date of birth), (b) binning (i.e., turning continuous values into categories), or (c) aggregating.Similarly, Cheshire (2009) overviews several recommendations to properly and safely anonymize qualitative data, such as using distal pseudonyms in place of real names, deleting identifying information such as interview locations, restricting access to anonymous transcripts only (e.g., no access to audio or other data formats), and digitally manipulating images or videos (e.g., disguising voices or blurring faces).O. Klein et al. (2018) provide additional guidance on how to deal with concerns related to participant privacy, whereas Rocher, Hendrickx, and de Montjoye (2019) address several limitations.A third solution to both privacy and copyright issues is non-consumptive data use (van Atteveldt et al., 2019), which means providing access to the data without physically copying it (e.g., by means of onsite visits).Fourth, researchers can share synthetic datasets.These are simulated data that "mimic real datasets by preserving their statistical properties and the relationships between variables" (Quintana, 2019, p. 2).In synthetic datasets, all individual cases are fictitious and novel, whereas the general properties of the variables remain the same (e.g., means, variance, and covariance).Synthetic datasets thereby enable others to reproduce the results of a study while guaranteeing the anonymity of participants.Synthetic data can be computed using open-source software such as the R package synthpop.Finally, data can be shared using licenses that legally restrict use, for example for scientific purposes only.Likewise, researchers can use services that limit access to specific users.In general, when sharing their data researchers should be as restrictive as necessary and as open as possible.Whether or not the sharing of data is possible always needs to be evaluated in the context of each individual research project.When data cannot be shared at all, researchers should provide explicit reasons in their manuscripts.
There also are concerns surrounding preregistration.What if researchers have learned a better way to analyze the data since they preregistered?What if they find something interesting that was not predicted?Again, it is a common misconception that preregistration restricts the ways in which researchers can analyze their data.Preregistration permits researchers to explore their data or to adapt their plans.The only condition is that all deviations from the preregistration need to be explained and that all additional analyses need to be labeled as exploratory.
Another concern is that reviewers, editors, or readers might use open science as a heuristic for high quality.Although sharing materials can be considered a necessary condition for high quality, it is not a sufficient one: "Transparency doesn't guarantee credibility.Transparency allows others to evaluate the credibility of your scientific claims" (Vazire, 2019, p. 20).Deviations from the preregistration in the final publication are common (Claesen, Gomes, Tuerlinckx, & Vanpaemel, 2019).As a result, studies employing open science practices need to be evaluated just as carefully as traditional studies.Open science practices are no panacea, and they cannot prevent intentional fraud.
With regard to replicability, there are communication phenomena that we might not expect to replicate because of changes in external factors.For example, a study on the relationship between one's number of Facebook followers and others' perceptions of one's social attraction (Tong, Van Der Heide, Langwell, & Walther, 2008) failed to replicate ten years later (Lane, 2018).This failed replication, however, can be attributed to shifts in Facebook users' orientations towards the platform.This example illustrates that replications can fail for different reasons, because of poor design or actual changes in the phenomenon of interest.Failed replications help us in designing new studies to assist with future investigations.That said, by using appropriate methods and sound theory we should aim to produce findings that are robust, sustainable, and likely to generalize across time, samples, and contexts.
Open science practices generally, and preregistration specifically, might lead to an unfamiliar publication process.Published studies will likely feature more mixed findings and null results and, thus, present less coherent "stories" (Giner-Sorolla, 2012).However, when studying human thoughts, behavior, or media content, there is always noise; hence, it is unlikely that we should repeatedly find coherent narratives (Giner-Sorolla, 2012) or large effects (Funder & Ozer, 2019).We believe that embracing a culture that values this challenge will advance Communication far more than a culture favoring simple narratives that do not replicate.
Other concerns revolve around increased costs and additional labor.Adequately powered studies require larger samples, which in turn require more resources.The additional documentation that accompanies open science practices is laborious and demands more careful planning and administration.Individuals adopting open science practices may be unable to publish their results as quickly as they are accustomed to.Together, early career researchers (ECRs) especially might be concerned that this will lead to a "thin" publication record.Regarding the publication system, implementing open science practices such as the TOP Guidelines mean that reviewers are expected to review supplementary material and to attempt to reproduce the results for themselves, which means additional labor.Whereas some of these concerns are justified, others are not.For example, preregistration does not lead to more work but instead front-loads processes that are otherwise addressed in later stages of a research project.When submitted as registered report, ECRs can list accepted-in-principle manuscripts on their CV, even before data collection.Furthermore, when conducted as registered reports, mixed and null findings are more likely to get published (Allen & Mehler, 2019), making it easier to plan projects because publication does not hinge on results.Registered reports currently also have a higher chance of acceptance (Chambers, 2019).To reduce the additional burden for reviewers, some journals have already implemented verification processes to ensure the reproducibility of analytical results. 20Similar to current practices with regard to plagiarism checks, the process is carried out by an independent institute instead of the reviewer.Overall, however, there is no denying that open science practices require us to increase our efforts.Perhaps the best answer to concerns about additional labor is normative.We as a field need to focus on research quality and not on publication quantity (Nelson, Simmons, & Simonsohn, 2012).Indeed, several solutions that we propose might lead to fewer submissions, but these submissions would be of higher quality.If we submit fewer papers, we have more time to review those of others, including data and materials.Open science practices should be acknowledged and incentivized by the entire field: It is the current and future hiring committees, grant agencies, and student research supervisors that will ultimately determine our norms, and whether or not the increased efforts help or harm individual careers.
Finally, the solutions we have presented are only a subset of several useful practices (for a list of additional solutions, see Online Appendix SE).Our list is not exhaustive: More work is needed to address challenges and opportunities for different research domains, such as qualitative research, computational methods, or hermeneutic approaches.In the spirit of a self-correcting science, as we collectively move towards more open science practices in Communication, the agenda will be revisited, challenged, and expanded.Not everyone might be able to immediately adopt all points of the agenda.Our agenda is not all or nothing: Even an incremental adoption will bring important benefits to our field, and some progress is better than no progress.To effectively address all concerns and to tailor initiatives to all members of the Communication community, we encourage surveys of Communication researchers regarding their opinions on open science, their hopes, and their concerns.Until then, a survey of German social scientists found that hopes concerning the benefits of open science significantly exceeded concerns (Abele-Brehm, Gollwitzer, Steinberg, & Scho ¨nbrodt, 2019)-a situation that likely also applies to Communication.

Conclusion
Several of our suggestions for open science practices imply substantial changes to the way we conduct research.They require learning new skills and jettisoning some of our old routines.So why should we care?Because from an ethical perspective, the values of open science practices are aligned with our societal function as scientists (Bowman & Keene, 2018;Merton, 1974).Open science practices provide the basis for collaboration, make results available to the community, and facilitate a culture which does not judge a study by its outcome, but by the quality of its theory and methods.They even boost public trust in our profession.The most important reason to adopt open science practices, however, is epistemic.We as Communication scholars aim to establish robust and reliable findings.Open science practices will produce more credible results, foster the integrity of our discipline and, ultimately, enhance our knowledge about Communication processes.

Supporting Information
This manuscript contains online supplementary material, which can be found here: https://osf.io/usy4t/.
It is possible to sign and thereby endorse the agenda.To sign the agenda and for the current list of signatories, see Online Appendix SA.
Please note: Oxford University Press is not responsible for the content or functionality of any supplementary materials supplied by the authors.Any queries (other than missing material) should be directed to the corresponding author for the article.

Table 1
Summary of the Open Science Agenda