Powerful computational tools and methods are becoming ubiquitous in academic research. However, with this increase in computational power and complexity comes increased responsibility to ensure robustness and reliability of research outputs. Reproducibility, the ability to reproduce reported results from their underlying data, computer code and reported methodology, is the minimum requirement for assessing such robustness.
To promote reproducibility and provide opportunity for researchers to engage with it in practice we’ve developed ReproHacks, one-day reproducibility hackathons where participants attempt to reproduce research from published code and data, usually on their own laptop. It is also an opportunity for researchers to help others learn from their work by submitting their papers, code and data for reproduction and review. However, the traditional format, while appropriate for the time budget and level of interest of most researchers, excludes the examination of the reproducibility of computationally intensive research.
As such, with support from the EPSRC, we have partnered with the University of Warwick to develop the first High Performance Computing ReproHack event format!. The aim of this extended event is for participants to reproduce computationally intensive published research from associated code and data on the Sulis Tier 2 HPC system and feedback their experiences to authors as well as the group of participants. In addition to practical experience of general research reproducibility, participants will gain a better understanding of the particulars of reproducible computational environments on HPC systems. The event also provides an opportunity to explore the reproducibility of computationally intensive research.
While the current event will be piloted with University of Warwick CDT students, we aim for the format developed and materials produced to form a prototype for future HPC ReproHack events.
We invite authors who would like feedback on the reproducibility of computationally intensive research to submit details of their paper, code and data for reproduction and review. You can submit your paper on the ReproHack Hub. To ensure your paper is associated with this event, please make sure to associate it with it during submission. If you would like to make your work available for future HPC ReproHacks, we recommend including an HPC tag. Please see our Author Guidelines for more information.
To accommodate the potential additional involvement, support and execution time required to complete the challenge, the event will run over 11 days, beginning with a launch session on the 21st of March 2022 and ending with a closing event on the 31st March 2022, with drop in support sessions held in between.
During the launch we will have a welcome and introduction to the event followed by a morning of talks and training sessions aiming to help prepare participants for reproducing. In the afternoon, participants will form teams around the papers they wish to tackle and begin attempting to reproduce the work on the Sulis HPC system with support from academic mentors and Research Software Engineers (RSEs).
While participants will be free to work on their papers in their own time over the next 10 days, four 2 hr drop in sessions will also be held where participants will be able to get support if they are stuck with any aspect of the challenge from their CDT mentors and RSEs.
At the closing celebratory event, participants will regroup to present their experiences and share lessons learnt. We will also hear from a selection of invited speakers.
|21st March||10am - 1pm||Welcome, Introduction & Training|
|1pm - 2pm||Lunch|
|2pm - 5pm||Initial ReproHack Session|
|22nd March||2pm - 4pm||Drop In support session|
|24th March||2pm - 4pm||Drop In support session|
|28th March||2pm - 4pm||Drop In support session|
|30th March||2pm - 4pm||Drop In support session|
|31st March||10am - 1pm||Closing Event|
This paper proposes a probabilistic planner that can solve goal-conditional tasks such as complex continuous control problems. The approach reaches state-of-the-art performance when compared to current deep reinforcement learning algorithms. However, the method relies on an ensemble of deep generative models and is computationally intensive. It would be interesting to reproduce the results presented in this paper on their robotic manipulation and navigation problems as these are very challenging problems that current reinforcement learning methods cannot easily solve (and when they do, they require a significantly larger number of experiences). Can the results be reproduced out-of-the-box with the provided code?