HPC Carpentry

Andy Turner: a.turner@epcc.ed.ac.uk
EPCC, The University of Edinburgh
30 July 2018

Slide content is available under under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This means you are free to copy and redistribute the material and adapt and build on the material under the following terms: You must give appropriate credit, provide a link to the license and indicate if changes were made. If you adapt or build on the material you must distribute your work under the same license as the original.
Note that this presentation contains images owned by others. Please seek their permission before reusing these images.

Built using reveal.js

reveal.js is available under the MIT licence

Coordinators/Maintainers:

Bob Freeman, Harvard, USA
Christina Koch, University of Wisconsin, Madison, USA
Alan O'Cais, Juelich Supercomputing Centre, Germany
Alex Razoumov, Compute Canada
Jeff Sessions, Compute Canada
John Simpson, Compute Canada
Daniel Smith, MolSSI, USA
Peter Steinbach, Scionics, Germany
Andy Turner, EPCC, UK

What is HPC Carpentry?

Goals

Apply the Carpentry model for HPC training

Open-source, community-designed training

Easy to adapt for different systems

Accessible to different audiences

Initial learner profile

I am not sure how HPC can help my research?
I do not have the basic technical skills to use HPC.
Just give me what I need to get on with my research on HPC.
What is an HPC system anyway? How does it differ from the computer I am already using? What about “the Cloud”?
How do I get help and/or help myself when I get stuck?
I feel unsure and out of my depth with HPC.
I do not understand the shared nature of HPC systems.

Topics

Why use HPC?
Login and transferring data
Understanding HPC jargon
Basic understanding of HPC architectures and practical implications for their use
File systems on HPC systems
In-terminal text editors
modules, environment, applications
The scheduler and writing job submission scripts
Shared system etiquette
Troubleshooting strategies

What about teaching the shell??

There has been a lot of discussion about this!

Whether this is needed or not depends on the particular audience
HPC Carpentry is using a modular approach to add flexibility (more later).

Challenges

System differences

Need to be easy to incorporate with minimal editing
Plan to do this automatically as much as possible
Set a configuration option once and it will be built with the correct setup
As the community grows, more system types will be available
(Could also be used to provide different research community flavours of the material.)

Range of learner experience

Wide range of different backgrounds
- Could be grouped by ability or research area
Flexibility required in topics…but need to maintain coherency and avoid maintenance overhead
Plan to use a modular approach with recommended configurations

Status and Plans

Original instances

hpc-intro

Produced by Compute Canada

hpc-in-a-day

Produced by Peter Steinbach

Topic	hpc-intro	hpc-in-a-day
bash Shell	✓	✗
Why HPC?	✓	✓
Login, data transfer	✓	✓
Software modules	✓	✓
Scheduler	✓	✓
File systems	✓	✓
Shared system etiquette	✓	✓
Parallelism and profiling	✗	✓

Modularisation

Half-day modules:

hpc-shell: Introduction to bash (based on hpc-intro)
hpc-intro: All other topics from hpc-intro
- Merge in relevant material from hpc-in-a-day
- Merge in contributions from CarpentryCon 2018
Others (from original Compute Canada material):
- hpc-python-novice: Introduction to using Python on HPC systems (based on hpc-python)
- hpc-python-pipeline: Using Python for computational pipelines on HPC systems (based on hpc-python)
Potential to add additional modules if other topics are required.

Planned initial release of this structure later in 2018

Automation

hpc-in-a-day includes technology to switch between different systems, login hosts and schedulers by setting config options
- Uses Jekyll (https://jekyllrb.com/) features
Plan to incorporate this into hpc-intro
Could expand this technology to allow automatic switching of other things, for example:
- Examples from particular research areas
- Different file system setups

Governance

Initial group of maintainers has been formed (still need to assign specific maintainers for each lesson)
Guidance on using HPC Carpentry lessons drafted
Policy on accepting contributions drafted