Hybrid MPI and OpenMP programming tutorial - Prace Training Portal
Hybrid MPI and OpenMP programming tutorial - Prace Training Portal Hybrid MPI and OpenMP programming tutorial - Prace Training Portal
Thread/Process Affinity (“Pinning”)• Highly OS-dependent system calls– But available on all systemsLinux: sched_setaffinity(), PLPA(see below) hwlocSolaris: processor_bind()Windows: SetThreadAffinityMask()…• Support for “semi-automatic” pinning in some compilers/environments– Intel compilers > V9.1 (KMP_AFFINITY environment variable)– Pathscale– SGI Altix dplace (works with logical CPU numbers!)– Generic Linux: taskset, numactl, likwid-pin (see below)• Affinity awareness in MPI librariesSeen on SUN Ranger slides– SGI MPT– OpenMPI– Intel MPI– …Hybrid Parallel ProgrammingSlide69/154Rabenseifner, Hager, JostWidely usable example: Using PLPAunder Linux!— skipped —Explicit Process/Thread Binding With PLPA on Linux:http://www.open-mpi.org/software/plpa/• Portable Linux Processor Affinity• Wrapper library for sched_*affinity() functions– Robust against changes in kernel API• Example for pure OpenMP: Pinning of threads#include ...#pragma omp parallel{#pragma omp critical{if(PLPA_NAME(api_probe)()!=PLPA_PROBE_OK) {cerr
Likwid Tool Suite• Command line tools for Linux:– works with standard linux 2.6 kernel– supports Intel and AMD CPUs– Supports all compilers whose OpenMP implementation is based onpthreads• Current tools:– likwid-topology: Print thread and cache topology(similar to lstopo from the hwloc package)– likwid-pin: Pin threaded application without touching code– likwid-perfCtr: Measure performance counters (similar to SGI‘s perfex orlipfpm tools)– likwid-features: View and enable/disable hardware prefetchers (Core2only at the moment)– likwid-bench: Low-level benchmark construction toollikwid-topology – Topology information• Based on cpuid information• Functionality:– Measured clock frequency– Thread topology– Cache topology– Cache parameters (-c command line switch)– ASCII art output (-g command line switch)• Currently supported:– Intel Core 2 (45nm + 65 nm)– Intel Nehalem– AMD K10 (Quadcore and Hexacore)–AMDK8Hybrid Parallel ProgrammingSlide73/154Rabenseifner, Hager, JostHybrid Parallel ProgrammingSlide74/154Rabenseifner, Hager, JostOutput of likwid-topologyCPU name: Intel Core i7 processorCPU clock: 2666683826 Hz*************************************************************Hardware Thread Topology*************************************************************Sockets: 2Cores per socket: 4Threads per core: 2-------------------------------------------------------------HWThread Thread Core Socket0 0 0 01 1 0 02 0 1 03 1 1 04 0 2 05 1 2 06 0 3 07 1 3 08 0 0 19 1 0 110 0 1 111 1 1 112 0 2 113 1 2 114 0 3 115 1 3 1-------------------------------------------------------------— skipped —likwid-topology continuedSocket 0: ( 0 1 2 3 4 5 6 7 )Socket 1: ( 8 9 10 11 12 13 14 15 )-------------------------------------------------------------*************************************************************Cache Topology*************************************************************Level: 1Size: 32 kBCache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 )-------------------------------------------------------------Level: 2Size: 256 kBCache groups: ( 0 1 ) ( 2 3 ) ( 4 5 ) ( 6 7 ) ( 8 9 ) ( 10 11 ) ( 12 13 ) ( 14 15 )-------------------------------------------------------------Level: 3Size: 8 MBCache groups: ( 0 1 2 3 4 5 6 7 ) ( 8 9 10 11 12 13 14 15 )-------------------------------------------------------------• … and also try the ultra-cool -g option!Hybrid Parallel ProgrammingSlide75/154Rabenseifner, Hager, JostHybrid Parallel ProgrammingSlide76/154Rabenseifner, Hager, Jost
- Page 1 and 2: Hybrid MPI & OpenMPParallel Program
- Page 3: Pure MPIHybrid Masteronlypure MPIon
- Page 6: — skipped —SUN: Running hybrid
- Page 11 and 12: Outline• Introduction / Motivatio
- Page 13 and 14: — skipped —Running the codeExam
- Page 15 and 16: Avoiding locality problems• How c
- Page 17: CCCCCCOpenMP OverheadThread synchro
- Page 21 and 22: Outline• Introduction / Motivatio
- Page 23 and 24: Numerical Optimization inside of an
- Page 25 and 26: Experiment: Matrix-vector-multiply
- Page 27 and 28: Comparison:MPI based parallelizatio
- Page 29 and 30: Memory consumptionCase study: MPI+O
- Page 31 and 32: MPI rules with OpenMP /Automatic SM
- Page 33 and 34: — skipped —Thread Correctness -
- Page 35 and 36: — skipped —Top-down - several l
- Page 37 and 38: — skipped —Remarks on MPI and P
- Page 39 and 40: Summary• This tutorial tried to-
- Page 41 and 42: References (with direct relation to
- Page 43 and 44: Further references• Matthias Hess
Thread/Process Affinity (“Pinning”)• Highly OS-dependent system calls– But available on all systemsLinux: sched_setaffinity(), PLPA(see below) hwlocSolaris: processor_bind()Windows: SetThreadAffinityMask()…• Support for “semi-automatic” pinning in some compilers/environments– Intel compilers > V9.1 (KMP_AFFINITY environment variable)– Pathscale– SGI Altix dplace (works with logical CPU numbers!)– Generic Linux: taskset, numactl, likwid-pin (see below)• Affinity awareness in <strong>MPI</strong> librariesSeen on SUN Ranger slides– SGI MPT– Open<strong>MPI</strong>– Intel <strong>MPI</strong>– …<strong>Hybrid</strong> Parallel ProgrammingSlide69/154Rabenseifner, Hager, JostWidely usable example: Using PLPAunder Linux!— skipped —Explicit Process/Thread Binding With PLPA on Linux:http://www.open-mpi.org/software/plpa/• Portable Linux Processor Affinity• Wrapper library for sched_*affinity() functions– Robust against changes in kernel API• Example for pure <strong>OpenMP</strong>: Pinning of threads#include ...#pragma omp parallel{#pragma omp critical{if(PLPA_NAME(api_probe)()!=PLPA_PROBE_OK) {cerr
Change language