Thank you for considering submitting your paper for review!
Assessing whether your paper is trully reproducible is extremely difficult until someone else actually tries to reproduce it. Not only can this pick up minor setup differences that you wouldn't be able to test on your own computer but it also opens the work up to feedback from a variety of experiences and perspectives. The feedback can help improve the submitted work but help guide how you produce future work.
More broadly, submitting your work and letting others work with it gives opportunity for the academic community to consider what form such associated research materials should take and how we want to be able to use them.
Two of the most common reasons for participants having difficulty with reproductions are:
For more information on what participants might look for during reviews, have a look at our participant guidelines
Ideally you will have a DOI associated with your paper when you submit your paper to the ReproHack Hub. Post publication would be fine but we believe the best time to submit your paper is at the preprint stage.
The central paper list has a search box which participants can use to filter papers. However we recommend making good use of tags. Use them to tag your paper with any tools used (e.g. programming languages like R or Python) and keywords identifying the paper domain.
The Hub is designed to accommodate submitting papers for specific events (for example a thematic event involving an institution or specific domain). Select the event you want to associate your paper with from the Event drop down menu on the submission form in the Paper details section. Once associated with an event, the paper will appear on the Associated Papers list on the event page.
If you choose to associate your paper with an event, you can also specify whether you want your paper associated solely with that event under the Review Availability section. If you chose to make it available for review only at the associated event, your paper will be open for review only during the event and for the week following the event end date. Otherwise it will also be listed on the central paper list and made available for general review.
Note that you can change any of this settings at anytime by editing your paper through the edit paper page. For example, if you wish to associate your paper with a new event, just select the new event form the Event drop-down menu.
Firstly, reproducibility scores are not an objective reflection of your paper. They are not stadardised in any formal way (although we are working on ways to do this) and importantly they encompass the familiarity, previous experiences and domain knowledge of reviewers. Functionally they are meant more as a signal to other participants, especially those looking for something easy to work on, so we do ask authors not to get too invested in them.
Having said that, we do find that papers that make it easy for participants with no experience in a given domain / tool tend to score highest. Often this comes down to either very user friendly automation and/or detailed documentation with clear instructions that make no assumptions on participants prior knowledge.
If you are still not happy with your reproducibility score, you have the following options:
Yes! We've added a comment section at the bottom of each review that is visible only to the authors and reviewers. You can use this section to have discussions about each review. Please remember to abide by the code of conduct in all your communications.
You can set reviews on your paper to be private in the Review Visibility section, by switching Make reviews public off. This will also hide the Mean Reproducibility Score from the paper listing.
There are two types of notifications you can receive an email about. The first is notification that a review has been received and the second that a comment has been posted to the review by a reviewer. You can set this notifications when you first submit your paper or edit them at any time on your edit paper page.
You can edit your paper's availability for review at any time. In the edit paper page, toggle the archive paper switch to change it's availability. This will not remove the paper from the database but will make it unavailable for further reviews. We do suggest you consider turning off notifications instead of archiving. If you change your mind at any time, you can also unarchive using the same toggle.
We accept papers that require proprietary software. The only hindrance is that we can't guarantee participants with licenses to the required software. We suggest to highlight any such software requirements in the What to focus on section of your paper submission.
The current ReproHack format involves participants joining events using their own computers. However we are working on an HPC ReproHack format to explore reproducibility of computationally intesing research so we welcome submissions in view of such an event. If you do submit a computationally intensive paper, please make sure to tag it with HPC
and supply any know details about computational requirements in the What to focus on section.
You can access a list of any materials you have submitted on your user profile page which you can access by clicking on your username on the navigation bar.