Establishing the Replicability and Generalizability of Multi-Study Longitudinal Research

Abstract Replication and cross-validation of research findings across independent longitudinal studies is essential for a cumulative science. However, the interplay between harmonization, replication, and generalizability of results across interdisciplinary longitudinal studies can present remarkable challenges. Careful interpretation of multistudy results must include consideration of the age, birth cohort, health, and education of individuals in the sample, measurements, the number and spacing of assessments, and rates of response and attrition. Placed in a broader historical (or future) context, we must consider the representativeness of population sampling, historical period, and analytic method in understanding the replicability and generalizability of findings. In a multistudy context, harmonization can be considered at levels of research question, statistical models, and measurements and can minimize some sources of cross-study variability. I will discuss the challenges and benefits of harmonization and the coordinated analysis approach used by the IALSA research network to achieve results from multi-study integrative research.

research network that conducts integrative analysis of longitudinal aging studies (IALSA), Hofer will describe challenges posed by multiple sources of heterogeneity in conducting coordinated analyses, and ways of handling these challenges to maximize reproducibility. Next, Mroczek will illustrate these issues by providing two examples of coordinated analyses. This talk will highlight design features that promote openness and transparency in conducting research on longitudinal data. Third, Lodi-Smith will provide practical guidance and examples on preregistering complex projects, strategies for transparently reporting deviations from preregistrations, considerations in sharing sensitive data, and tips on transparent documentation of analysis code. She will also emphasize the pedagogical value of preregistration. Finally, Seaman will describe ongoing efforts to establish open science practices as the default in her laboratory, with the goal of providing a model for both junior and more established researchers wanting to build transparency into their research practices. Discussant Isaacowitz, editor for the Journal of Gerontology, Series B: Psychological Sciences, will evaluate the presentations from the lens of how journals can encourage more transparent and replicable scientific practices.

. Northwestern University Feinberg School of Medicine, Evanston, Illinois, United States
The application of openness and transparency principles is challenging when using existing or ongoing long-term longitudinal data. One technique that promotes replicability and also is consistent with openness and transparency principles is coordinated analysis. Such analyses, especially when done with a large number of extant longitudinal datasets, tend to draw upon values of data sharing, revelation of code and scripts, and pre-registration. Thus coordinated analyses often provide good examples of how multiple transparency and openness values can come together. We will demonstrate this by presenting two recent large-scale coordinated analyses. One was a 15-study investigation of personality and mortality risk (Graham et al., 2017). The second is a new 16-study investigation of personality trajectories (Graham et al., under revision). We show how multi-study designs are congruent with open science and transparency ideas in the context of longitudinal and other secondary data.

ESTABLISHING THE REPLICABILITY AND GENERALIZABILITY OF MULTI-STUDY LONGITUDINAL RESEARCH Scott Hofer, University of Victoria, Victoria, British Columbia, Canada
Replication and cross-validation of research findings across independent longitudinal studies is essential for a cumulative science. However, the interplay between harmonization, replication, and generalizability of results across interdisciplinary longitudinal studies can present remarkable challenges. Careful interpretation of multistudy results must include consideration of the age, birth cohort, health, and education of individuals in the sample, measurements, the number and spacing of assessments, and rates of response and attrition. Placed in a broader historical (or future) context, we must consider the representativeness of population sampling, historical period, and analytic method in understanding the replicability and generalizability of findings. In a multistudy context, harmonization can be considered at levels of research question, statistical models, and measurements and can minimize some sources of cross-study variability. I will discuss the challenges and benefits of harmonization and the coordinated analysis approach used by the IALSA research network to achieve results from multi-study integrative research.

APPLIED OPEN SCIENCE FOR SECONDARY DATA ANALYSIS AND META-ANALYSIS Jennifer Lodi-Smith, Canisius College, Buffalo, New York, United States
This talk will provide guidance on the practicalities of open science for secondary data analysis and meta-analyses. Example studies will provide practical considerations for preregistering complex projects, insights into strategies for transparently reporting deviations from preregistrations, advice on deciding when and how to share sensitive data, and tips on transparent documentation of analysis code. Examples will be drawn from an ongoing meta-analysis of the relationship between self-concept clarity and self-esteem (https://osf.io/sa2bx/), the Rochester Adult Longitudinal Study (https://osf.io/ya4ph/), and the Aging and Autism Study (https://osf.io/g9c3e/). The pedagogical value of preregistration will be emphasized throughout the talk.

MAKING OPEN SCIENCE THE DEFAULT: CREATING LAB PRACTICES TO PROMOTE TRANSPARENCY Kendra Seaman, The University of Texas at Dallas, Dallas, Texas, United States
Many factors disincentivize researchers, particularly junior faculty members, from implementing open science practices. One way to make open science less burdensome is integrate open science methods with existing procedures. I will describe my ongoing efforts to establish open science practices as the default in my laboratory. These strategies include (1) creating and updating a lab manual to set expectations for openness, (2) articulating a standard operating procedure for creating, preregistering, and managing a new project, (3) establishing clear organizational structures for data, code, and data products, and (4) training lab members on the use of these and other open science tools like GitHub. These strategies provide a model for both junior researchers starting a lab and more established researchers who want to build transparency into their research practices. Ultimately, implementing open science methods will improve lab workflows and improves the overall quality of our science.