Review of
"REMoDNaV: robust eye-movement classification for dynamic stimulation"

Review of "REMoDNaV: robust eye-movement classification for dynamic stimulation"

Submitted by erlija  

Aug. 31, 2022, 9:20 a.m.

Lead reviewer

erlija

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
6 / 10
Briefly describe the procedure followed/tools used to reproduce it

The code to reproduce the work was acquired via git clone. Running make, the final pdf file could be created. Due to some fonts missing, figure 1 looked slightly different.

It was also tested to run make after make clean, to recalculate all results, create all figures and the final pdf file. This did not run as smoothly as expected. Firstly, git annex gives some error as data files are not available when running datalad get. Apparently some data is present in data, and the analysis is partly conducted. Due to lacking knowledge of the datalad functionality, I was not able to identify the source of those errors. Latexmk did not create the bbl file needed to produce the pdf file, the Makefile was changed to create this file (added option -bibtex). Due to the above errors, the script stops when running the mk_event_duration_histograms function. The following functions are never called, therefore the kappa values are not computed and not written to results_def.tex. This results in errors when running pdflatex.

Briefly describe your familiarity with the procedure/tools used by the paper.

I am familiar with python but have not used datalad or git-annex before. I have no domain knowledge and am not familiar with the applied methods.

Which type of operating system were you working in?
Linux/FreeBSD or other Open Source Operating system
What additional software did you need to install?

I needed to install git-annex, this dependency is not given in the README.md

What software did you use

I used python 3.10. The used python version to run this code were not given by the authors. Since the version numbers for e.g. pandas in the Makefile are not compatible with python 3.10, I had to modify the versions. I used pandas 1.4.3 instead of 1.0.5.

What were the main challenges you ran into (if any)?

Since I am not familiar with datalad, I was not able to identify the errors. It is unclear whether this is a problem with the code, datalad or just a wrong path.

What were the positive features of this approach?

I would assume, that this script should easily work, if the data could be downloaded using datalad get.

Any other comments/suggestions on the reproducibility approach?

Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

The README.me could also name git-annex as a requirement. The used python version should be mentioned. Due to some errors I looked more closed into the code. As I am not familiar with the research, more comments would have helped me to better understand the code.

What do you like about the documentation?

A README.md is given. Most function have a docstring, which is nice.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
5 / 10
Any suggestions on how the analysis could be made more transparent?

Reusability

Reusability rating
Rate the project on reusability of the material
5 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

A license file could be helpful.



Any final comments

Thank you for making your research available!