Review of
"Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study"

Review of "Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study"

Submitted by DianneSvdS  

Oct. 12, 2023, 11:58 a.m.

Lead reviewer


Review Body


Did you manage to reproduce it?
Fully Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
10 / 10
Briefly describe the procedure followed/tools used to reproduce it

As a researcher at a non-academic institute I cannot use Code Ocean to run the code. Instead, I created a project in Rstudio using the Posit workbench and downloaded everything needed using git.

Briefly describe your familiarity with the procedure/tools used by the paper.

I am not familiar with Code Ocean, but very familiar with R.

Which type of operating system were you working in?
Cloud Computing
What additional software did you need to install?

Nothing else needed to install.

What software did you use


What were the main challenges you ran into (if any)?

A project file needed to be created in the /code folder otherwise the 'here' statements did not work. After that most of the code worked except for this part:

{r render_appendix, include=FALSE} render_appendix("supplementary_information.Rmd") the Papaya::render_appendix is no longer supported. Quite a few changes needed to be made to get everything running smoothly again;

At the top of the .Rmd file the output statement has to change to:

output : pdf_document

The render_apendix statement was adjusted as follows:



Appendix A

```{r appendix : "supplementary_information.Rmd"}


However, even though this produces all information needed, nicely in one paper, the original paper has the appendix in a separate document.

What were the positive features of this approach?

Figures and the main text of the paper were produced without problems and look the same as the original paper.

Any other comments/suggestions on the reproducibility approach?

Try to make the project reproducible also without Code Ocean, perhaps also provide a Docker container.


Documentation rating
How well was the material documented?
10 / 10
How could the documentation be improved?

Nicely documented, perhaps include a line in the Readme file that if you download the project, you need to put the project file in the /code folder instead of the main folder.

What do you like about the documentation?

I have not read in detail but it looks alright

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
7 / 10
Any suggestions on how the analysis could be made more transparent?

all code is there publicly available, as well as the data and the paper


Reusability rating
Rate the project on reusability of the material
6 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

Provide a Docker container as well or try to improve the project in such a way that you can also download it and reproduce it in R/Rstudio.

Any final comments

Thanks, it was a nice learning experience for me :)