Recommendations to enhance rigor and reproducibility in biomedical research


 Biomedical research depends increasingly on computational tools, but mechanisms ensuring open data, open software, and reproducibility are variably enforced by academic institutions, funders, and publishers. Publications may present software for which source code or documentation are or become unavailable; this compromises the role of peer review in evaluating technical strength and scientific contribution. Incomplete ancillary information for an academic software package may bias or limit subsequent work. We provide 8 recommendations to improve reproducibility, transparency, and rigor in computational biology—precisely the values that should be emphasized in life science curricula. Our recommendations for improving software availability, usability, and archival stability aim to foster a sustainable data science ecosystem in life science research.

Abstract: Computational methods have reshaped the landscape of modern biology. While the biomedical community is increasingly dependent on computational tools, the mechanisms ensuring open data, open software, and reproducibility are variably enforced by academic institutions, funders, and publishers. Publications may present academic software for which essential materials are or become unavailable, such as source code and documentation. Publications that lack such information compromise the role of peer review in evaluating technical strength and scientific contribution. Incomplete ancillary information for an academic software package may bias or limit any subsequent work produced with the tool. We provide eight recommendations across four different domains to improve reproducibility, transparency, and rigor in computational biology-precisely the main values which should be emphasized in life science curricula. Our recommendations for improving software availability, usability, and archival stability aim to foster a sustainable data science ecosystem in biomedicine and life science research.
"Snakemake (https://snakemake.readthedocs.io/en/stable/, RRID:SCR_003475) is a tool to create reproducible and scalable data-analyses workflows, with a language based on python. Snakemake makes it easier to execute data analyses on different environments without modification on the workflow definition." Editor's Comment #5: F1000 had more basic R/ggplot2 examples since 2014 (https://blog.f1000.com/2014/09/09/living-figures-interview/), and has had examples with Plotly since 2017. This is more sophisticated, but if you want to keep this example you should tweak the wording so it sounds less like a first. Technically it was also a demonstration of a reproducible view of an existing eLife article. I would suggest something like: In 2018, eLife published a demonstration of a dynamic and code-based reproducible peer-reviewed paper, using the Stencila platform and Binder.
Authors' Response: We have changed the text according to the Editor's suggestion.
"In 2018, eLife published a demonstration of a dynamic and code-based reproducible peer-reviewed paper, using the Stencila platform and Binder (Table 1). This approach enables data and analysis to be fully reproducible by the reader and challenges the traditional static representation of results using PDF or HTML formats." Editor's Comment #6: Need to remove this final reference and just include the URL. Can include this in the Acknowledgements section or an Editor's Note if that fits better. And as the paper is forkable/contributable to by others do you want to encourage others to contribute?.
Authors' Response: We have moved the referred statements to the Acknowledgements section. We have also removed the reference and added the URL at the end of the Conclusion section as follows.
"Given this is a fast-moving area, some of our recommendations are likely to be outdated within a short period and other short-lived. We acknowledge that new platforms may appear soon (https://github.com/Mangul-Lab-USC/enhancing_reproducibility)."

"Acknowledgments
We thank Dr. Lana Martin for helping with the design of Figure 1 and constructive discussions and comments on the manuscript. A dynamic version of this paper reflects new platforms https://github.com/Mangul-Lab-USC/enhancing_reproducibility, as an example to command scientific rigor and reproducibility ourselves. This dynamic version was compiled in markdown and includes an extended list of references. We encourage others to contribute to our repository." Editor's Thank you very much for considering our paper, entitled " Recommendations to enhance rigor and reproducibility in biomedical research, " for publication at GigaScience . We are grateful to the reviewers for their careful assessment of the manuscript and the detailed and constructive comments that they provided. We believe that addressing these comments produced a much stronger manuscript that will hold more value for the scientific community.
Following the editor's comments, we have made changes listed below as a point-by-point response. We have also removed the reference and added the URL at the end of the Conclusion section as follows.
" Given this is a fast-moving area, some of our recommendations are likely to be outdated within a short period and other short-lived. We acknowledge that new platforms may appear soon ( https://github.com/Mangul-Lab-USC/enhancing_reproducibility ). "

" Acknowledgments
We thank Dr. Lana Martin for helping with the design of Figure 1 and constructive discussions and comments on the manuscript. A dynamic version of this paper reflects new platforms https://github.com/Mangul-Lab-USC/enhancing_reproducibility , as an example to command scientific rigor and reproducibility ourselves. This dynamic version was compiled in markdown and includes an extended list of references. We encourage others to contribute to our repository. " Editor's Comment #7: Need the article number/DOI, so ideally should cite: