Review of
"The viewing angle in AGN SED models, a data-driven analysis"

First we ran the code via the included binder. Then we tried to delete rawdata and download them and re-create figures from scratch. We also tried to install the package locally.

Different for each reviewer

• Ilja's not familiar with python or jupyter. Transferrable skills from R, org-mode, GNU/Linux.
• Harry: Faniliar with python, a little with jupyter, zero experience with physics modules
• Sheeba familiar with python, Jupyter, and Binder but zero experience with physics modules
• Igor: familiar with Python and Jupyter notebook, but not Binder and used physics Python packages.

To run locally, we needed to install:

• git
• jupyterlab
• anaconda
• and then install the environment packages with conda env create -f environment.yml

See above.

Notebook 5 is intentionally missing, but this was unclear from the binder. Lack of master script linking notebook code. Unable to reproduce data without raw (deleted original data and tried to reproduce from pulldown)

Great use of the binder framework for sharing the code and documentation, we loved the pre-rendered html files. Setting it up as a binder that doesn't require the user to install anything locally was a great way to make the analysis quickly reproducible for everyone! The fact that the files were neatly split into separate analysis chunks was useful, and the step-by-step file provided a great overview linking everything together.

We didn't find the binder button immediately, so it was not obvious that we could inspect everything in an online pre-configured environment. There were no guidelines on how to run the analysis locally. Which commands do you need to run to install the dependencies via the envornment.yml file? How do I start the jupyter lab after installing the dependencies locally?

Figuring out which codebook does what, and why. The README nicely describes all the tools that were used, but did not include a big-picture description of what problem the project is solving. The step-by-step was a nice file, but lacked the more general introduction as well.

• It would be nice to 0-pad the notebook filenames so that they are sorted correctly by default.
• Adding a master file that runs the different codebooks would be nice.
• It could perhaps be merged with the step-by-step file.
• Furthermore, this would make it more clear that the code for step 5 is not in this repo. It would also be possible to add it to the repo, but set all the code chunks to not evaluate, or to provide a warning that running this part of the analysis requires HPC.
• Each notebook could provide more details on the goal and background of the notebook.
• It was not immediately clear how to run the environment (we found the binder button only after some time). The hyperlink in the text points to binder's homepage.

Documentation was good, they've shown how each script should be pieced together with a brief description of each step

• We enjoyed the step-by-step document, the way the analysis is split into separate notebooks with a single task. The use of autopep8 in the notebooks for formatting the code is good.
• We like the inclusion of rendered html files and intermediate output, which allowed us to quickly inspect everything without needing to run the chunks.

Very transparent paper/code, no corners have been cut or hidden from the public view

• Subdirectories inside Data/{Raw,Interim,Final,Complementary} directories, e.g. Data/Interim/CIGALEOutputs, must be created while the sripts are running, in case user deletes them.
• Hard-coded filenames with dates Notebook 2_Clean_Sample.ipynb: Cell 3
• Some galaxies are empty causing value errors, if-statements should be applied to prevent this Notebook 2_Clean_Sample.ipynb: Cell 15
• Switch to semantic versioning
• Add a Changelog

We'd like to thank the authors for putting so much effort into making their workflow reproducible! They went above and beyond, and the comments above are mostly nitpicks that prevent the workflow from being fully reproducible from scratch.