Hybrid MPI and OpenMP programming tutorial - Prace Training Portal

Content• Thread-safetyqualityofMPIlibraries........... 120– MPI rules with OpenMP 121– Thread support of MPI libraries 124– Thread Support within OpenMPI 125• Tools for debugging and profiling MPI+OpenMP . . 126– Intel ThreadChecker 127– Performance Tools Support for Hybrid Code 129• Other options on clusters of SMP nodes ........133– Pure MPI – multi-core aware 134– Hierarchical domain decomposition 135– Scalability of MPI to hundreds of thousands 140– Remarks on Cache Optimization 141– Remarks on Cost-Benefit Calculation 142– Remarks on MPI and PGAS (UPC & CAF) 143• Summary...................................147– Acknowledgements 148– Summaries 149– Conclusions 154• Appendix...................................155– Abstract 156– Authors 157– References (with direct relation to thecontent of this tutorial) 160– Further references 165• Content ....................................172PracticalitiesAbout compilationRemember to enable OpenMP when compiling your OpenMP and hybridprograms! With PGI by having the –mp flag, with GNU the –fopenmp flagand so on.Pure OpenMP jobsYou can run OpenMP enabled codes on a multicore (Linux) laptop simplyby setting the environment variable OMP_NUM_THREADS equal to thenumber of threads you wish to execute, e.g. ”exportOMP_NUM_THREADS=8”, and running the program as usual.Louhi is not meant as a production platform for flat-OpenMP codes, but fortraining and testing purposes you can utilize Louhi as well. There anadditional -d option to the aprun launcher must be set and the number ofMPI processes must be set to one:aprun –n 1 –d 8 ./my_omp.exeHybrid MPI+OpenMP codesHybrid programs are executed in the systems combining the execution ofa MPI job and an OpenMP one. On the Cray systems, the executioncommandaprun –n 4 –d 8 ./my_hyb.exeWould launch an interactive job with 4x8=32 cores (4 MPI tasks eachhaving 8 threads).Hybrid Parallel ProgrammingSlide 173 / 154Rabenseifner, Hager, JostWhen executing through the batch job system, additional lines (ascompared to a flat MPI program) to the batch script are needed; cf. thissample jobscript#!/bin/bash#PBS -l walltime=00:15:00#PBS -l mppwidth=16#PBS -l mppdepth=8#PBS -l mppnppn=1cd $PBS_O_WORKDIRexport OMP_NUM_THREADS=8aprun -n 16 -N 1 -d 8 ./my_hyb.exeThis would allocate and execute a 16x8=128 core job.HybridprogrammingexercisesJacobi iterationThe Jacobi iterator is a way of solving the Poisson equation 2 V= byiteratively update the value of a 2D array V asVnew(i,j)=[V(i-1,j)+V(i+1,j)+V(i,j-1)+V(i,j+1)-(i,j)]/4Until convergence has been reached (i.e. V new and V old are sufficientlyclose to each other).HybridprogrammingexercisesJacobi solver hybridizedSee the Jacobi solvers parallelized with MPI based on 1D decompositionof the array (jacobi-mpi.f90 or .c). Starting from that program,implement an MPI+OpenMP hybrid Jacobi solver with three differentrealizations:1. Fine-grained version, where the halo exchange is performed outsidethe parallel region. This basically means just wrapping the updatesweep in a parallel do construct.2. Version where the master thread carries out the halo exchange, andall threads are alive throughout the program execution.3. Version employing multiple thread communication.Parallel Jacobi solverThe update of each element requires only information from the nearestneighbors, therefore if the whole domain can be decomposed to paralleltasks (in either one or two dimensions). Only the boundaries need to becommunicated, and one (MPI) task needs to communicate only with two(1D decomposition) or four (2D decomposition) other tasks. Thereforethe Jacobi solver isdo while (!converged)communicate boundaries (”exchange”)make the update above (”sweep”)check convergenceend doGet acquainted with the Jacobi solvers parallelized with MPI based on 1Ddecomposition of the array (jacobi-mpi.f90 or .c). The domain isdecomposed in row-wise (C) or column-wise blocks (Fortran) with theindex limits (which depend on the number of MPI tasks and the size ofthe domain) computed in the procedure “decomp”. Note that twoiterations are performed in one cycle of the update loop (to enable reuseof the arrays and more convenient checking for the convergence).Note, that the hybridization here essentially corresponds to a 2Ddecomposition.Sample answers are provided in files jacobi-hyb-a…c.f90 or .c.Cray specific remarkWhen using the Cray systems, you may need to adjust the thread safetyof the MPI library with the MPICH_MAX_THREAD_SAFETY variable, e.g.setenv MPICH_MAX_THREAD_SAFETY serializedor inserting a lineexport MPICH_MAX_THREAD_SAFETY=serializedto the batch job script. The minimum thread safety level is ”single” forExercise 1, ”funneled” for Exercise 2 and ”multiple” for Exercise 3.In addition, you will need to link with -lmpich_threadm for multiplethread safety support.