Review of
"REMoDNaV: robust eye-movement classification for dynamic stimulation"

Review of "REMoDNaV: robust eye-movement classification for dynamic stimulation"

Submitted by LarissaKo  

July 1, 2024, 4:22 p.m.

Lead reviewer

LarissaKo

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
9 / 10
Briefly describe the procedure followed/tools used to reproduce it
  • created virtual environment
  • installed DataLad (v1.0.2), git-annex (v10.20240430), DataLad Containers extension package (v1.2.5) and Docker (v26.0.0)
    • for Docker, the Subscription Service Agreement need's to be accepted
  • cloned repository from datalad
  • run datalad rerun results-containerized'
    • got output:
      text action summary: add (ok: 5) get (notneeded: 2, ok: 483) install (ok: 1) run (ok: 1) save (notneeded: 3, ok: 2) unlock (notneeded: 2)
  • installed Inkscape v1.3.2 and texlive-latex-extra to view the latex document (I installed them because I had problems at first, but I'm not sure if they were actually needed)
  • run datalad containers-run -n docker-make main.pdf
    • output:
      text action summary: get (notneeded: 1) run (ok: 1) save (notneeded: 5)
Briefly describe your familiarity with the procedure/tools used by the paper.
  • some experience with command line
  • familiar with Python
  • no experience with DataLad or Docker
  • not familiar with eye-movement capturing
Which type of operating system were you working in?
Apple Operating System (macOSX)
What additional software did you need to install?

Appart from the things mentioned in the README:

  • git-annex
What software did you use

Only command line

What were the main challenges you ran into (if any)?

When I first tried it I had trouble viewing the results. After cloning the repository once more and recreating the steps it worked just fine.

What were the positive features of this approach?
  • very easy reproducable
  • everything worked after installing all the software and running some commands
  • easy installation for macOS
Any other comments/suggestions on the reproducibility approach?

I would have liked more explanation as to what the output (action summary) meant.


Documentation

Documentation rating
How well was the material documented?
7 / 10
How could the documentation be improved?
  • git-annex should be added to the dependencies
  • the algorithm is explained in the paper, but it would be nice to also have some information about how the code works in the README
  • some more information about what the output means would be nice.
What do you like about the documentation?
  • the setup instructions are easy to find and follow
  • all mandatory and optional parameters are well described
After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
4 / 10
Any suggestions on how the analysis could be made more transparent?

There was no need to look at the code or methods at all to get the results. It also was not explained in the README or elsewhere. This makes it easy reproducible, but doesn't help to get familiar with how it works.


Reusability

Reusability rating
Rate the project on reusability of the material
9 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?


Any final comments