Review of
"Revisiting the zonally asymmetric extratropical circulation of the Southern Hemisphere spring using complex empirical orthogonal functions"

Review of "Revisiting the zonally asymmetric extratropical circulation of the Southern Hemisphere spring using complex empirical orthogonal functions"

Submitted by rcaneill  

March 13, 2024, 1:16 p.m.

Lead reviewer

rcaneill

Review team members

acocac

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
1 / 10
Briefly describe the procedure followed/tools used to reproduce it

We used the provided docker image with R. We then installed the R packages, but some were missing so we had to install them by hand. We also had to change the working directory:

setwd('shceof')

Missing R packages: Installing R Packages: 'bookdown', 'ggnewscale', 'ggpattern', 'ggperiodic', 'ggrepel', 'ppcor', 'shadowtext', 'widyr'

Missing R package: tagger

devtools::install_github("eliocamp/tagger") > requires devtools

Briefly describe your familiarity with the procedure/tools used by the paper.

1 of the 2 reviewer used R long time ago, the 2nd one never used R.

Which type of operating system were you working in?
Linux/FreeBSD or other Open Source Operating system
What additional software did you need to install?

We had to install docker to use the docker container.

What software did you use

Docker, Rstudio

What were the main challenges you ran into (if any)?

The main challenges were:

  1. Installing all the dependencies
  2. Understanding the R jargon (e.g. Knit a file)
  3. Getting to download the data. From what we understood data should be downloaded automatically, but we did not manage to get it work
What were the positive features of this approach?

The README file describes how to install the package and provides the commands to type. It seems that dependencies versions are locked, which is good for reproducibility.

Any other comments/suggestions on the reproducibility approach?

Maybe providing a command to type in a terminal to reproduce the results and paper would be helpful for people that do not have knowledge of Rstudio, but who are familiar with a terminal.


Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

For me (rcaneill) who do not have experience in Rstudio, I was lost in where to type which command.

What do you like about the documentation?

The authors took time to write the different steps to recompile the document and reproduce the figures. They provide two different methods to install Rstudio: either via apt on the global Ubuntu-like system, or via docker container. The container is very convenient as it does not pollute the rest of the system.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
1 / 10
Any suggestions on how the analysis could be made more transparent?

Include more details of the workflow in the README file i.e. which is the path to the manuscript (Rmd file) and how to set up dependencies and inputs (where to store them) to reproduce results (figures, tables, etc). Also, we suggest documenting workarounds for potential errors e.g. use install.package() for missing dependencies.


Reusability

Reusability rating
Rate the project on reusability of the material
5 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

No licence is provided for the code.



Any final comments

We spent roughly 1 hour on trying to run the code, which is not very long. We believe that with 1) a little bit more experience in Rstudio and 2) some more time we would have been able to reproduce the results. The fact that the paper itself is written in R-markdown (and hence executes the code at compilation time) gives confidence in the reproducibility of the code itself.