Powerful computational tools and methods are becoming ubiquitous in academic research. However, with this increase in computational power and complexity comes increased responsibility to ensure robustness and reliability of research outputs. Reproducibility, the ability to reproduce reported results from their underlying data, computer code and reported methodology, is the minimum requirement for assessing such robustness.
To promote reproducibility and provide opportunity for researchers to engage with it in practice we’ve developed ReproHacks, one-day reproducibility hackathons where participants attempt to reproduce research from published code and data, usually on their own laptop. It is also an opportunity for researchers to help others learn from their work by submitting their papers, code and data for reproduction and review. However, the traditional format, while appropriate for the time budget and level of interest of most researchers, excludes the examination of the reproducibility of computationally intensive research.
As such, with support from the EPSRC, we have partnered with the University of Warwick to develop the first High Performance Computing ReproHack event format!. The aim of this extended event is for participants to reproduce computationally intensive published research from associated code and data on the Sulis Tier 2 HPC system and feedback their experiences to authors as well as the group of participants. In addition to practical experience of general research reproducibility, participants will gain a better understanding of the particulars of reproducible computational environments on HPC systems. The event also provides an opportunity to explore the reproducibility of computationally intensive research.
While the current event will be piloted with University of Warwick CDT students, we aim for the format developed and materials produced to form a prototype for future HPC ReproHack events.
We invite authors who would like feedback on the reproducibility of computationally intensive research to submit details of their paper, code and data for reproduction and review. You can submit your paper on the ReproHack Hub. To ensure your paper is associated with this event, please make sure to associate it with it during submission. If you would like to make your work available for future HPC ReproHacks, we recommend including an HPC tag. Please see our Author Guidelines for more information.
To accommodate the potential additional involvement, support and execution time required to complete the challenge, the event will run over 11 days, beginning with a launch session on the 21st of March 2022 and ending with a closing event on the 31st March 2022, with drop in support sessions held in between.
During the launch we will have a welcome and introduction to the event followed by a morning of talks and training sessions aiming to help prepare participants for reproducing. In the afternoon, participants will form teams around the papers they wish to tackle and begin attempting to reproduce the work on the Sulis HPC system with support from academic mentors and Research Software Engineers (RSEs).
While participants will be free to work on their papers in their own time over the next 10 days, four 2 hr drop in sessions will also be held where participants will be able to get support if they are stuck with any aspect of the challenge from their CDT mentors and RSEs.
At the closing celebratory event, participants will regroup to present their experiences and share lessons learnt. Lunch will also be provided. The closing event will be held at Space 30 of the Radcliffe Conference Centre.
Directions: From the main entrance follow the corridor to the left and follow signs for Space 30 (Map : Space 30 shows up as MR9)
Folk are welcome to arrive early and help themselves to coffee and nibbles in the main lounge, but we’ll also have our own coffee area immediately adjacent to the conference room.
|21st March||10:00 - 10:15||Anna Krystalli: Welcome, Introduction to event.|
|10:15 - 10:40||Martin Callaghan: HPC in the age of Data Science|
|10:40 - 11:30||Twin Karmakharm: Reproducibility on HPC|
|11:30 - 11:45||COFFEE|
|11:45 - 12:10||Heather Ratcliffe: Introduction to Sulis|
|12:10 - 12:35||Mike Croucher: Reproducible MATLAB on HPC|
|12:35 - 12:55||Mozhgan Kabiri Chimeh: NVIDIA AI & HPC Containers|
|13:00 - 14:00||LUNCH|
|14:00 - 14:20||Anna Krystalli: Introduction to ReproHacking|
|14:20 - 17:00||Initial ReproHack Session|
|22nd March||14:00 - 16:00||Drop In support session|
|24th March||14:00 - 16:00||Drop In support session|
|28th March||14:00 - 16:00||Drop In support session|
|30th March||14:00 - 16:00||Drop In support session|
|31st March||10:00 - 13:00||Closing Event|
This paper proposes a probabilistic planner that can solve goal-conditional tasks such as complex continuous control problems. The approach reaches state-of-the-art performance when compared to current deep reinforcement learning algorithms. However, the method relies on an ensemble of deep generative models and is computationally intensive. It would be interesting to reproduce the results presented in this paper on their robotic manipulation and navigation problems as these are very challenging problems that current reinforcement learning methods cannot easily solve (and when they do, they require a significantly larger number of experiences). Can the results be reproduced out-of-the-box with the provided code?
There are many applications to multi-MeV X-rays. Their penetrative properties make them good for scanning dense objects for industry, and their ionising properties can destroy tumours in radiotherapy. They are also around the energy of nuclear transitions, so they can trigger nuclear reactions to break down nuclear waste into medical isotopes, or to reveal smuggled nuclear-materials for port security. Laser-driven X-ray generation offers a compact and efficient way to create a bright source of X-rays, without having to construct a large synchrotron. To fully utilise this capability, work on optimising the target design and understanding the underlying X-ray mechanisms are essential. The hybrid-PIC code is in a unique position to model the full interaction, so its ease-of-use and reproducibility are crucial for this field to develop.
The results of this paper have been used in multiple subsequent studies as a benchmark against which other methods of performing the same calculation have been tested. Other groups have challenged the results as suffering from finite size effects, in particular the calculations on mixtures of cubic and hexagonal ice. Should there be time during in the event, participants could check this by performing calculations on larger unit cells. Each individual calculation should converge adequately within 96 hours making it amenable to a HPC ReproHack. Given modern HPC hardware many such calculations could be run concurrently on a single HPC node.
The negative surface enthalpies in figure 5 are surprising. At least one group has attempted to reproduce these using a different code and obtained positive enthalpies. This was attributed to the inability of that code to independently relax the three simulation cell vectors resulting in an unphysical water density. This demonstrates how sensitive these results can be to the particular implementation of simulation algorithms in different codes. Similarly the force field used is now very popular. Its functional form and full set of parameters can be found in the literature. However differences in how different simulation codes implement truncation, electrostatics etc can lead to significant difference in results such as these. It would be a valuable exercise to establish if exactly the same force field as that used here can be reproduced from only its specification in the literature. The interfacial energies of interest should be reproducible with simulations on modest numbers of processors (a few dozen) with run times for each being 1-2 days. Each surface is an independent calculation and so these can be run concurrently during the ReproHack.
DFT calculations are in principle reproducible between different codes, but differences can arise due to poor choice of convergence tolerances, inappropriate use of pseudopotentials and other numerical considerations. An independent validation of the key quantities needed to compute electrical conductivity would be valuable. In this case we have published our input files for calculating the four quantities needed to parametrise the transport simulations from which we compute the electrical conductivity. These are specifically electronic band structure, phonon dispersions, electron-phonon coupling constants and third derivatives of the force constants. Each in turn in more sensitive to convergence tolerances than the last, and it is the final quantity on which the conclusions of the paper critically depend. Reference output data is provided for comparison at the data URL below. We note that the pristine CNT results (dark red line) in figure 3 are an independent reproduction of earlier work and so we are confident the Boltzmann transport simulations are reproducible. The calculated inputs to these from DFT (in the case of Be encapsulation) have not been independently reproduced to our knowledge.
Popular descriptors for machine learning potentials such as the Behler-Parinello atom centred symmetry functions (ACSF) or the Smooth Overlap of Interatomic Potentials (SOAP) are widely used but so far not much attention has been paid to optimising how many descriptor components need to be included to give good results.
Systematically improvable machine learning potentials could have a significant impact on the range of properties that can be modelled, but the toolchain associated with using them presents a barrier to entry for new users. Attempting to reproduce some of our results will help us improve the accessibility of the approach.
We do care about reproducibility. In case we receive any feedback, we would be really happy to improve our Github repository so as to make the reproduction easier!
We do care about reproducibility. In case we receive any feedback, we would be really happy to improve our Github repository and/or submitted manuscript so as to make the reproduction easier!
This paper presents a fine example of high-throughput computational materials screening studies, mainly focusing on the carbon nanoclusters of different sizes. In the paper, a set of diverse empirical and machine-learned interatomic potentials, which are commonly used to simulate carbonaceous materials, is benchmarked against the higher-level density functional theory (DFT) data, using a range of diverse structural features as the comparison criteria. Trying to reproduce the data presented here (even if you only consider a subset of the interaction potentials) will help you devise an understanding as to how you could approach a high-throughput structure prediction problem. Even though we concentrate here on isolated/finite nanoclusters, AIRSS (and other similar approaches like USPEX, CALYPSO, GMIN, etc.,) can also be used to predict crystal structures of different class of materials with applications in energy storage, catalysis, hydrogen storage, and so on.
The method is trained on the data that were available, but it is meant to be re-trainable as soon as new data are published. It would be great to be really sure that even someone else will be able to do it. In case we receive any feedback, we would be really happy to improve our Github repository so as to make the reproduction easier!
The direct numerical simulations (DNS) for this paper were conducted using Basilisk (http://basilisk.fr/). As Basilisk is a free software program written in C, it can be readily installed on any Linux machine, and it should be straightforward to then run the driver code to re-produce the DNS from this paper. Given this, the numerical solutions presented in this paper are a result of many high-fidelity simulations, which each took approximately 24 CPU hours running between 4 to 8 cores. Hence the difficulty in reproducing the results should mainly be in the amount of computational resources it would take, so HPC resources will be required. The DNS in this paper were used to validate the presented analytical solutions, as well as extend the results to a longer timescale. Reproducing these numerical results will build confidence in these results, ensuring that they are independent of the system architecture they were produced on.
This work is well cited in the field of laser driven ion acceleration and provides a good study on the interaction of multiple laser driven ion acceleration mechanisms along with the impact of relativistic induced transparency.