Review of
"REMoDNaV: robust eye-movement classification for dynamic stimulation"

Review of "REMoDNaV: robust eye-movement classification for dynamic stimulation"

Submitted by alubitz  

Nov. 10, 2022, 2:48 p.m.

Lead reviewer

alubitz

Review team members

rdk

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
7 / 10
Briefly describe the procedure followed/tools used to reproduce it

Running on Ubuntu 20.04 and 22.04

  • create virtual env with virtualenv on Ubuntu 22.04 and with anaconda on Ubuntu 20.04
  • Installing dependencies
  • clone repository
  • run make
    • This builds the paper
  • run make clean
  • rerun make
    • This failed on Ubuntu 22.04 (error: failed while downloading numpy package dependencies)
    • This failed on Ubuntu 20.04 (error: RuntimeError: main thread is not in main loop)
Briefly describe your familiarity with the procedure/tools used by the paper.
  • Very familiar with python
  • Somehow familiar with make files
  • not familiar with datalad
Which type of operating system were you working in?
Linux/FreeBSD or other Open Source Operating system
What additional software did you need to install?

inkscape as mentioned in the dependencies

What software did you use

vs code

What were the main challenges you ran into (if any)?

errors poped up after running make after make clean.

What were the positive features of this approach?

One could see that the authors had reproducibility in mind when creating the plots

Any other comments/suggestions on the reproducibility approach?

Dependencies should be mentioned before the make command.


Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

It seems data download is more than 550MB

What do you like about the documentation?

It is very clean. Maybe there could be more details in other md-files for the interested reader.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
8 / 10
Any suggestions on how the analysis could be made more transparent?

The make file abstracts a lot which is great for a fast reproduction but there could be more info on how to run the python scripts for the interested reader. And a list of direct links to the different datasets and their description could be helpful.


Reusability

Reusability rating
Rate the project on reusability of the material
8 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

In the readme display which dataset are used.



Any final comments

In general great work. It is visible that you were thinking about reproducibility thoroughly while creating the plots. The version in gitpod does not work and could easily be fixed using a different container with dependencies installed.