Review of
"REMoDNaV: robust eye-movement classification for dynamic stimulation"

Review of "REMoDNaV: robust eye-movement classification for dynamic stimulation"

Submitted by Einstrick  

June 26, 2024, 1:49 p.m.

Lead reviewer


Review Body


Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
8 / 10
Briefly describe the procedure followed/tools used to reproduce it

I just followed the instructions presented in the ReadMe file within the GitHub given by the authors. There are 2 approaches presented, an outdated and a new approch; I followed the new approach and even though there were some difficulties regarding Docker on Windows, in the end it worked out fine. At first I had to install the newest version of Python via, followed by downloading and installing git for windows and git-annex for windows. Next step, I downloaded and installed DataLad via their Website and the container extension package via pip command. In the end, I downloaded and installed the Docker software and ran the commands presented in the ReadMe file to clone the repository und rerun it with Datalad. There was a minor problem, at first I had to manually save a Datalad file via the command % datalad save in order to create a clean file and working tree. After that I rerun the repository.

Briefly describe your familiarity with the procedure/tools used by the paper.

I used GitHub in the past, but I never used any of the other presented software/tools.

Which type of operating system were you working in?
Windows Operating System
What additional software did you need to install?
  1. Newest Python
  2. Git for Windows
  3. Git-annex for Windows
  4. DataLad for Windows
  5. DataLad container extersion package for Windows
  6. Docker for Windows
What software did you use

GitHub, DataLad and Docker I ran all commands on Windows CMD

What were the main challenges you ran into (if any)?

There were some difficulties regarding Docker on Windows which is well known within the Docker-community. In gerenal it is recommended to run Docker on a linux based operating system but in the end it ran on my operating system as well. One might have to use a VM and make sure that virtualization is ensured in BIOS, which can be tough to find but it is managable. In addition, the authors stated, that in case of full reproducibility, no change to the dataset will be saved (indicated by save (notneeded)). In my case, there were 5 changes indicated. Furthermore I had tons of warnings, that certain keyword arguments won't be supported in future, so I think the reproducibility won't be guaranteed anymore and time soon.

What were the positive features of this approach?

It is easy to follow the instructions in general, but it can be tough to fix problems occuring with the software for people who didn't work with Datalad or git-annex yet

Any other comments/suggestions on the reproducibility approach?

As stated in the challenges part, I think the code has to be revised due to literally hundreds of warning I got during the execution of the command.


Documentation rating
How well was the material documented?
9 / 10
How could the documentation be improved?

I have no suggestions for improvement.

What do you like about the documentation?

The repository is very well organized and documented.

After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
7 / 10
Any suggestions on how the analysis could be made more transparent?

Not at all.


Reusability rating
Rate the project on reusability of the material
9 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?

It could be marked which author worken on which part of the project / code since everybody has its own style of coding and it would be interesting whether everyone aligned their style or if one revised the whole code and aligned it.

Any final comments

Overall i enjoyed reviewing the paper very much, I made first experiences with container-based evironments and the advantagea are clear to me. Thank you very much for submitting the paper to Reprohack,