Review of
"Neurodesk: an accessible, flexible and portable data analysis environment for reproducible neuroimaging"

Review of "Neurodesk: an accessible, flexible and portable data analysis environment for reproducible neuroimaging"

Submitted by abannachbrown  

June 30, 2024, 2:10 p.m.

Lead reviewer

abannachbrown

Review team members

jankaWIS

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
4 / 10
Briefly describe the procedure followed/tools used to reproduce it

We followed the extensive instructions to install the NeuroDesk, desktop version on two computers, one Mac (Sonoma 14.5) and one Windows (Windows 11 64-bit operating system, x64-based processor). We were able to install the desktop version and we tried to reproduce the analysis in the paper by following the instructions on: https://osf.io/e6pw3/wiki/home/ We tried to reproduce the analyses in the paper, however as this was a one day hackathon and the code takes an estimated 2 days, we were unable to complete the code running. While running the code we had to change a couple of directory paths to get it to run.

Briefly describe your familiarity with the procedure/tools used by the paper.

We were not familar with NeuroDesk and had some previous experience with neuro image processing and basic programming.

Which type of operating system were you working in?
Windows Operating System
What additional software did you need to install?

Docker & NeuroDesk

What software did you use

We installed the desktop version on NeuoDesk on both Windows and Mac. (And Docker and WSL Ubuntu for Windows)

What were the main challenges you ran into (if any)?

It's unclear where and what the data are at https://ida.loni.usc.edu/ as that opens only the general page. There is a supplementary file associated with the article that is not on OSF. Is it the same data? We were only able to download the two .nii files from the online version of the article (https://static-content.springer.com/esm/art%3A10.1038%2Fs41592-023-02145-x/MediaObjects/41592_2023_2145_MOESM4_ESM.zip). Though some of our ReproHack team members had issues accessing this data because the article was not open access.

What were the positive features of this approach?

We appreciate the extensive documentation for installing the NeurDesk software on Neurodesk.org. In particular, the “Set-up page” which automatically detects the user's system and device specifications was particularly useful.

Any other comments/suggestions on the reproducibility approach?

Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

At the Installation of NeuroDesk, in “Step 2 - Run NeuroDesk app”, Option 2 “Advanced with Terminal”. Here it was unclear what is the difference between option A and option B (with or without a persistent home directory) and why a user should choose between one at the other. https://www.neurodesk.org/docs/getting-started/neurodesktop/windows/ Also at this installation stage (https://www.neurodesk.org/docs/getting-started/neurodesktop/mac/), it would be very useful to the user, to describe what the code is actually doing so the user known what is going on in the docker. In the tutorial about mriqc (https://www.neurodesk.org/tutorials-examples/tutorials/functional_imaging/mriqc/), it's extensive with many screenshots etc, but one way to make it even better, would be to show an example on a provided dataset. It's a bit unclear what and where the data are and how to work with the path within NeuroDesk.

What do you like about the documentation?

On NeuroDesk.org website, we really liked all the screenshots and code that could directly be copy pasted into the user's own terminal. Well done for a really nice Software Platform!

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
5 / 10
Any suggestions on how the analysis could be made more transparent?

Explain the rationale behind some of the installation steps would make it even more accessible for the user.


Reusability

Reusability rating
Rate the project on reusability of the material
3 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

We were not sure whether the data included in the Nature article had a permissive license as it was copyrighted by the authors and the journal. The OSF page also did not have a license attahed to it.



Any final comments