Review of
"REMoDNaV: robust eye-movement classification for dynamic stimulation"

Review of "REMoDNaV: robust eye-movement classification for dynamic stimulation"

Submitted by vronsee  

Aug. 31, 2022, 9:59 a.m.

Lead reviewer

vronsee

Review Body

Reproducibility

Did you manage to reproduce it?
Fully Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
8 / 10
Briefly describe the procedure followed/tools used to reproduce it

I cloned the GitHub Repo to my local machine. I installed the dependencies mentioned in the readme file. Then I executed the make file. At first it didn't work and I had to delete the version numbers of python packages in the make file. In a second try I also had to download git-annex which wasn't mentioned in the dependencies. Then it worked.

Briefly describe your familiarity with the procedure/tools used by the paper.

I have basic python knowledge and good command line skills. I have never executed a make file before.

Which type of operating system were you working in?
Apple Operating System (macOSX)
What additional software did you need to install?

The dependencies mentioned in the readme file and git-annex.

What software did you use

Python and the command line.

What were the main challenges you ran into (if any)?

The make file didn't run correctly because of different versions of python and python packages. I had to delete the version numbers of python packages in the makefile.

What were the positive features of this approach?

The approach with the make file is very user friendly and nice because you only have to execute one file and few commands to reproduce the results.

Any other comments/suggestions on the reproducibility approach?

It would have been nice to add the git-annex to the dependencies in the readme file. Also the used python version would have been good to avoid incompatibility issues. A container, a conda environment or a detailed list of python version and python package versions used would be a nice solution.


Documentation

Documentation rating
How well was the material documented?
9 / 10
How could the documentation be improved?

Include the git-annex dependency.

What do you like about the documentation?

Clear instructions on how to execute the code.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
8 / 10
Any suggestions on how the analysis could be made more transparent?

Reusability

Reusability rating
Rate the project on reusability of the material
10 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

License and citation could be included in the readme file.



Any final comments