HPC ReproHack @ Warwick

HPC ReproHack @ Warwick

      March 21, 2022, 10 a.m. - March 31, 2022, 1 p.m. (Europe/London)

   University of Warwick, Coventry, CV4 7AL, United Kingdom

Hosted by: University of Warwick


Event submitted by: annakrystalli on Jan. 5, 2022, 2:25 p.m.

Event Description

Powerful computational tools and methods are becoming ubiquitous in academic research. However, with this increase in computational power and complexity comes increased responsibility to ensure robustness and reliability of research outputs. Reproducibility, the ability to reproduce reported results from their underlying data, computer code and reported methodology, is the minimum requirement for assessing such robustness.

To promote reproducibility and provide opportunity for researchers to engage with it in practice we’ve developed ReproHacks, one-day reproducibility hackathons where participants attempt to reproduce research from published code and data, usually on their own laptop. It is also an opportunity for researchers to help others learn from their work by submitting their papers, code and data for reproduction and review. However, the traditional format, while appropriate for the time budget and level of interest of most researchers, excludes the examination of the reproducibility of computationally intensive research.

As such, with support from the EPSRC, we have partnered with the University of Warwick to develop the first High Performance Computing ReproHack event format!. The aim of this extended event is for participants to reproduce computationally intensive published research from associated code and data on the Sulis Tier 2 HPC system and feedback their experiences to authors as well as the group of participants. In addition to practical experience of general research reproducibility, participants will gain a better understanding of the particulars of reproducible computational environments on HPC systems. The event also provides an opportunity to explore the reproducibility of computationally intensive research.

While the current event will be piloted with University of Warwick CDT students, we aim for the format developed and materials produced to form a prototype for future HPC ReproHack events.

Paper Submissions

We invite authors who would like feedback on the reproducibility of computationally intensive research to submit details of their paper, code and data for reproduction and review. You can submit your paper on the ReproHack Hub. To ensure your paper is associated with this event, please make sure to associate it with it during submission. If you would like to make your work available for future HPC ReproHacks, we recommend including an HPC tag. Please see our Author Guidelines for more information.

Event Format

To accommodate the potential additional involvement, support and execution time required to complete the challenge, the event will run over 11 days, beginning with a launch session on the 21st of March 2022 and ending with a closing event on the 31st March 2022, with drop in support sessions held in between.

Launch Day

During the launch we will have a welcome and introduction to the event followed by a morning of talks and training sessions aiming to help prepare participants for reproducing. In the afternoon, participants will form teams around the papers they wish to tackle and begin attempting to reproduce the work on the Sulis HPC system with support from academic mentors and Research Software Engineers (RSEs).

Drop In Support Sessions

While participants will be free to work on their papers in their own time over the next 10 days, four 2 hr drop in sessions will also be held where participants will be able to get support if they are stuck with any aspect of the challenge from their CDT mentors and RSEs.

Closing Event

At the closing celebratory event, participants will regroup to present their experiences and share lessons learnt. We will also hear from a selection of invited speakers.

Agenda

Date Time Event
21st March 10am - 1pm Welcome, Introduction & Training
1pm - 2pm Lunch
2pm - 5pm Initial ReproHack Session
22nd March 2pm - 4pm Drop In support session
24th March 2pm - 4pm Drop In support session
28th March 2pm - 4pm Drop In support session
30th March 2pm - 4pm Drop In support session
31st March 10am - 1pm Closing Event

Associated papers

  • PlanGAN: Model-based Planning With Sparse Rewards and Multiple Goals

    Authors: Henry Charlesworth and Giovanni Montana
    DOI: None
    Submitted by gmontana74      

    Why should we attempt to reproduce this paper?

    This paper proposes a probabilistic planner that can solve goal-conditional tasks such as complex continuous control problems. The approach reaches state-of-the-art performance when compared to current deep reinforcement learning algorithms. However, the method relies on an ensemble of deep generative models and is computationally intensive. It would be interesting to reproduce the results presented in this paper on their robotic manipulation and navigation problems as these are very challenging problems that current reinforcement learning methods cannot easily solve (and when they do, they require a significantly larger number of experiences). Can the results be reproduced out-of-the-box with the provided code?