Review of
"Use of significance test logic by scientists in a novel reasoning task"

Review of "Use of significance test logic by scientists in a novel reasoning task"

Submitted by betty_syriopoulou  

May 5, 2022, 1:28 p.m.

Lead reviewer

betty_syriopoulou

Review Body

Reproducibility

Did you manage to reproduce it?
Fully Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
10 / 10
Briefly describe the procedure followed/tools used to reproduce it

We reproduced the manuscript with R version 4.0.3 and 4.0.5 (2021-03-31) and R studio Version 1.4.1106 (Windows 10)

Briefly describe your familiarity with the procedure/tools used by the paper.

We were 3 people with various levels of familiarity of Rmarkdown.

Which type of operating system were you working in?
Windows Operating System
What additional software did you need to install?

None

What software did you use

We reproduced the manuscript with R version 4.0.3 and 4.0.5 (2021-03-31) and R studio Version 1.4.1106 (Windows 10)

What were the main challenges you ran into (if any)?

It would be nice to have an overview in the README file about the descriptions and organisation of the subfolders, for example, where manuscript.Rmd is located. It was a bit difficult to find the main file(manuscript.rmd) but once we did it was easy to go ahead.

Could consider a solution like renv to “freeze” dependency versions, as failures are more likely to happen as time goes and dependencies are further developed.

What were the positive features of this approach?

Nicely organized with a specific function for each manuscript figure! We also liked the info about compilation versions at the end of the manuscript!

Any other comments/suggestions on the reproducibility approach?

Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

Include info about the structure of the folders and what each one includes so that it is easier to navigate.

What do you like about the documentation?

Each file included enough details about the content.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
8 / 10
Any suggestions on how the analysis could be made more transparent?

Reusability

Reusability rating
Rate the project on reusability of the material
9 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?


Any final comments

There were three of us doing this review: Yuliya Leontyeva, Robert Karlsson, Betty Syriopoulou Thank you for sharing the material, in general it has been a very nice experiment and the package was well made. :) Really loved the background story! The idea of creating an article in the form of a package can be a template for other reproducible articles, especially for R users.