Review of
"Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study"

Review of "Analytic reproducibility in articles receiving open data badges at the journal Psychological Science: An observational study"

Submitted by ccamara  

Nov. 18, 2021, 4:11 p.m.

Lead reviewer


Review team members

ccamara betty_syriopoulou yulya

Review Body


Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
9 / 10
Briefly describe the procedure followed/tools used to reproduce it
  • We cloned/download the repository;
  • We had to create a RStudio project (no project was found in the repo). This was necessary to make relative paths work (without it, all calls to here() will not work);
  • We opened manuscript.Rmd, as prompted in the description here:;
  • We had to manually Install a bunch of dependencies. We will elaborate on some suggestions for improvement later on;
  • We run all the chunks (which worked well once the previous steps were sorted out);
  • At that point, we could knit the manuscript (manuscript.Rmd);
  • We had some issues with the {papaja} package, which was not available from CRAN and we had to install from GitHub (install.packages() won't work). Then, we had to dig up an older version (installed with devtools::install_github("crsh/papaja@1999ba3") to actually manage to knit to pdf;
  • The previous step was successful on macOS and Linux; on two separate university-managed Windows laptops, that did not succeed.
  • Some warnings appeared here and there, but that didn't affect the output.
Briefly describe your familiarity with the procedure/tools used by the paper.

Our group had different degrees of familiarity with R and R markdown. Users with less familiarity felt that a more descriptive README file would have been useful to identify where to start, how the repository was organised, and so on.

Which type of operating system were you working in?
Apple Operating System (macOSX)
What additional software did you need to install?

We also used a laptop with a rolling Ubuntu-based Linux distribution with a KDE Neon desktop environment, and two university-managed Windows 10 laptops. We needed to install/update R and a variety of R packages, which ended up being somewhat frustrating. The {papaja} and {meta} packages gave us the biggest headache, especially {papaja} which was not on CRAN at the moment of our ReproHacking (2021-11-18).

What software did you use

R and RStudio, a variety of R packages (hard dependencies), pandoc/LaTeX to knit to pdf.

What were the main challenges you ran into (if any)?

The main challenges we found were: - Sorting out dependencies. This was the main issue; - Understanding how the material was organised: we felt that having a more descriptive/comprehensive README file would have simplified out effort; - We couldn't test the CodeOcean stuff without subscribing first; - It would have been useful if the Dockerfile from the CodeOcean page had been included in the Data&Analysis repository, as it fully specifies the dependencies of the project.

What were the positive features of this approach?

Once every dependency (and paths) were sorted out, we could just run the analysis and knit the pdf. That was super easy.

Any other comments/suggestions on the reproducibility approach?

The structure of the data&analysis repository was very good. Once we sorted out dependencies, it was super easy for us to reproduce all the results of the paper, so good job for that! In just a couple of hours of work we could sort everything out and achieve all of the above.

Some further thoughts, which we felt would have simplified our effort ever more:

  • The manuscript contained several links to different sub-folders (?) on OSFHome, which was a bit confusing. Maybe a single URL, e.g. to the homepage of the project, would be easier to follow?
  • Less experienced users would have benefit from a more informative README file (and possibly even the authors if they need to revise the project after some time).


Documentation rating
How well was the material documented?
4 / 10
How could the documentation be improved?
  • The README file could be improved to better tie all the material together;
  • We think this is the first file a person checks out when opening a repository, and here (at least, in the data/materials repo) we found little guidance;
  • It was not evident to us what the different components archived on OSFHome are, how the different datasets are tied together, and for us, with no previous exposure to this project, it was (at times) hard to figure out what we were looking at. This could be covered in the README file, possibly?
  • We did not realise the whole amount of work behind this project; the authors should be more proud of this and showcase it better to readers and ReproHackers :)
What do you like about the documentation?

Each single R script is very well documented, actually, so we really liked that. However, at a "macro" level, it was hard for us to tie together the different components/scripts of this project if we wanted to dig more deeply into the various aspects. Again, please refer to our previous comments on a global README file.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
6 / 10
Any suggestions on how the analysis could be made more transparent?

We didn't really have a lot of time to go through the code and methods, but we felt that everything was ok. All R code was in separate scripts, so we did not really have to check that to recompile the document.


Reusability rating
Rate the project on reusability of the material
8 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

The MIT license did not have copyright year or holder. We're not sure how this affect the possibility to re-use code.

Any final comments

We hope our comments don't come across as harsh, we think the authors did a good job to structure the project and ensure the results and manuscript are fully reproducible. There were no obvious shortcomings, so we focussed on the details! :) Thanks for sharing this paper with the ReproHack community, this is a nice "meta" example on reproducibility, and we enjoyed working on it.