HPC Carpentry

Andy Turner: a.turner@epcc.ed.ac.uk
EPCC, The University of Edinburgh
16 September 2019

Slide content is available under under a Creative Commons Attribution-NonCommercial-ShareAlike 4.0 International License.

This means you are free to copy and redistribute the material and adapt and build on the material under the following terms: You must give appropriate credit, provide a link to the license and indicate if changes were made. If you adapt or build on the material you must distribute your work under the same license as the original.
Note that this presentation contains images owned by others. Please seek their permission before reusing these images.

Built using reveal.js

reveal.js is available under the MIT licence

What is HPC Carpentry?

Goals

Apply the Carpentry model for HPC training

Open-source, community-designed training

Easy to adapt for different systems

Accessible to different audiences

Initial learner profile

I am not sure how HPC can help my research?
I do not have the basic technical skills to use HPC.
Just give me what I need to get on with my research on HPC.
What is an HPC system anyway? How does it differ from the computer I am already using? What about “the Cloud”?
How do I get help and/or help myself when I get stuck?
I feel unsure and out of my depth with HPC.
I do not understand the shared nature of HPC systems.

Topics

Why use HPC?
Login and transferring data
Understanding HPC jargon
Basic understanding of HPC architectures and practical implications for their use
File systems on HPC systems
In-terminal text editors
modules, environment, applications
The scheduler and writing job submission scripts
Shared system etiquette
Troubleshooting strategies

What about teaching the shell??

There has been a lot of discussion about this!

Whether this is needed or not depends on the particular audience
HPC Carpentry is using a modular approach to add the flexibility required (more later).

Challenges

System differences

Need to be easy to incorporate with minimal editing
Plan to do this automatically as much as possible
Set a configuration option once and it will be built with the correct setup
As the community grows, more system types will be available
(Could also be used to provide different research community flavours of the material.)

Range of learner experience

Wide range of different backgrounds
- Could be grouped by ability or research area
Flexibility required in topics…but need to maintain coherency and avoid maintenance overhead
Plan to use a modular approach with recommended configurations

Status and Plans

Original instances

hpc-intro

Produced by Compute Canada

hpc-in-a-day

Produced by Peter Steinbach

Topic	hpc-intro	hpc-in-a-day
bash Shell	✓	✗
Why HPC?	✓	✓
Login, data transfer	✓	✓
Software modules	✓	✓
Scheduler	✓	✓
File systems	✓	✓
Shared system etiquette	✓	✓
Parallelism and profiling	✗	✓

Modularisation

Half-day modules:

Available: hpc-shell: Introduction to bash (based on hpc-intro)
Available: hpc-intro: All other topics from hpc-intro
- Merge in relevant material from hpc-in-a-day
- Merge in contributions from CarpentryCon 2018
Others planned (from original Compute Canada material):
- hpc-python-novice: Introduction to using Python on HPC systems (based on hpc-python)
- hpc-python-pipeline: Using Python for computational pipelines on HPC systems (based on hpc-python)
- Tentative: hpc-parallel: Introduction to parallel methods and concepts, task farm, shared memory, message passing, basic benchmarking/profiling
Others exising:
- hpc-chapel: Introduction to the Cray Chapel PGAS langauge (Compute Canada lesson)
Potential to add additional modules if other topics are required.

Automation

hpc-in-a-day includes technology to switch between different systems, login hosts and schedulers by setting config options
- Uses Jekyll/Liquid (https://jekyllrb.com/) features
Currently working to incorporate this into hpc-intro (should be available by end Oct 2019)
Could expand this technology to allow automatic switching of other things, for example:
- Examples from particular research areas
- Different file system setups