Hybrid MPI and OpenMP programming tutorial - Prace Training Portal

Intra-node MPI characteristics: IMB Ping-Pong benchmark• Code(toberunon2processors):wc = MPI_WTIME()do i=1,NREPEATif(rank.eq.0) thenMPI_SEND(buffer,N,MPI_BYTE,1,0,MPI_COMM_WORLD,ierr)MPI_RECV(buffer,N,MPI_BYTE,1,0,MPI_COMM_WORLD, &status,ierr)elseP PMPI_RECV(…)C CCMPI_SEND(…)endifChipsetenddowc = MPI_WTIME() - wc• Intranode (1S): mpirun –np 2 –pin “1 3” ./a.out• Intranode (2S): mpirun –np 2 –pin “2 3” ./a.out• Internode: mpirun –np 2 –pernode ./a.outHybrid Parallel ProgrammingSlide61/154Rabenseifner, Hager, JostMemoryPCPCCIMB Ping-Pong: LatencyIntra-node vs. Inter-node on Woodcrest DDR-IB cluster (Intel MPI 3.1)Latency [μs3,532,521,510,50Hybrid Parallel ProgrammingSlide62/1543,24Rabenseifner, Hager, Jost0,55P PC CC0,31IB internode IB intranode 2S IB intranode 1SChipsetMemoryP PC CCAffinity matters!IMB Ping-Pong: Bandwidth CharacteristicsIntra-node vs. Inter-node on Woodcrest DDR-IB cluster (Intel MPI 3.1)OpenMP OverheadP PC CCChipsetMemoryP PC CCHybrid Parallel ProgrammingSlide63/154Between two cores ofone socketRabenseifner, Hager, Jostintranodeshm commBetween two socketsof one nodeShared cacheadvantageBetween two nodesvia InfiniBandAffinity matters!• As with intra-node MPI, OpenMP loop start overhead varies with themutual position of threads in a team• Possible variations– Intra-socket vs. inter-socket– Different overhead for “parallel for” vs. plain “for”– If one multi-threaded MPI process spans multiple sockets,• … are neighboring threads on neighboring cores?• … or are threads distributed “round-robin” across cores?• Test benchmark: Vector triad#pragma omp parallelfor(int j=0; j < NITER; j++){#pragma omp (parallel) forfor(i=0; i < N; ++i)a[i]=b[i]+c[i]*d[i];if(OBSCURE)dummy(a,b,c,d);}Hybrid Parallel ProgrammingSlide64/154Rabenseifner, Hager, JostLook at performance for smallarray sizes!