Review of
"Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA"

Review of "Measuring the impact of COVID-19 vaccine misinformation on vaccination intent in the UK and USA"

Submitted by PTiringer  

June 30, 2024, 3:10 p.m.

Lead reviewer

PTiringer

Review team members

JohannesF99

Review Body

Reproducibility

Did you manage to reproduce it?
Partially Reproducible
Reproducibility rating
How much of the paper did you manage to reproduce?
7 / 10
Briefly describe the procedure followed/tools used to reproduce it
  • In the first step, we downloaded the repository and read the paper to get into the topic itself.
  • Next we read the Documentation, README and Jupyter Notebooks and briefly inspected all directories and containing files.
  • Afterward, we created a virtual python environment and tried to run every code cell in the notebooks as good as possible with PyCharm Professional 2023.2.5 and Python 3.12.3
  • When we encountered errors due to missing dependencies, we tried to resolve them by installing them, which worked in most cases.
  • Additionally, we compared the given information with the content in the actual paper.
Briefly describe your familiarity with the procedure/tools used by the paper.
  • We already worked with Jupyter Notebooks in a few other university courses, so we were more or less familiar with the tool stack, what really helped us to get started.
  • Some of the used python packages were also familiar to us, but a few others like pystan were completely unknown to us.
Which type of operating system were you working in?
Linux/FreeBSD or other Open Source Operating system
What additional software did you need to install?
  • Software needed for this project is some kind of IDE and a python interpreter, which we already had installed on our systems. Additional software was not needed, just the python dependencies used in the project.
  • Those dependencies were:
    • matplotlib
    • pandas
    • pyreadstat
    • seaborn
    • pystan
What software did you use
  • PyCharm Professional 2023.2.5
  • Python 3.12.3
  • git/GitHub
What were the main challenges you ran into (if any)?
  • One of the main challenges were the missing raw data files the project depended on. A portion of the notebooks couldn't be executed, because they depend on the processing of these files.
  • Another challenge was the usage of the pystan package in the project. Because of major API changes in the newer versions of the package, the code in the project was not able to run. When trying to run the notebooks with the current version of pystan and a few code changes, we managed to run a few more lines of code, but eventually ended up with another error, we couldn't resolve without rewriting the whole code that depends on pystan.
What were the positive features of this approach?
  • Using a fully-fledged IDE took most of the configuration like the creation of a virtual environment from us, so we could focus pretty fast on the actual code. Furthermore, PyCharm has built-in support for Jupyter Notebooks, what really helped, too.
Any other comments/suggestions on the reproducibility approach?
  • We really liked the reproducibility approach, learned a lot, and are likely to use this tech stack in the future for similar projects again.

Documentation

Documentation rating
How well was the material documented?
8 / 10
How could the documentation be improved?

-

What do you like about the documentation?
  • They had one short documentation where they described the structure of the project, which gave a good overall summary. From there, we could inspect all directories and had a pretty good basic understanding of what the content is about.
  • Also the use of Jupyter Notebooks is always a good sign for documentation, because you can combine documentation with code, we always knew why the authors would do something and what the code does.
After attempting to reproduce, how familiar do you feel with the code and methods used in the paper?
6 / 10
Any suggestions on how the analysis could be made more transparent?
  • We think they did a pretty good job with the analysis.
  • What we really liked is, that they did not include too much medical information in the code or documentation, because that would distract the reader when coming from a non-medical background.

Reusability

Reusability rating
Rate the project on reusability of the material
7 / 10
Permissive Data license included:  
Permissive Code license included:  

Any suggestions on how the project could be more reusable?
  • We think they did a pretty good job with the analysis.
  • What we really liked is, that they did not include too much medical information in the code or documentation, because that would distract the reader when coming from a non-medical background.


Any final comments