Parallel version - CSC

Tête Rousse in ParallelSchäfer May 2011 1Martina Schäferfer, , Arctic Centre, UoL

steps parallel mesh changes to the sif run the simulation Martina Schäfer May 2011 2

Martina Schäfer May 2011 3

create mesh in 6 partitionsmesh (1)ElmerGrid 2 2 WithCavity –halo –metis 6 4elmer input formatmesh folder# procMartina Schäfer May 2011 4

mesh (2) after the simulation to connect the single .ep filestogetherElmerGrid 15 3 WithCavity/teterousse_step3Martina Schäfer May 2011 5

mesh (3)Martina Schäfer May 2011 6

Martina Schäfer May 2011 7

sif (1) direct method umfpack does not work in parallel use mumps instead replace Linear System Direct Method = umfpackwith Linear System Direct Method = mumps in Solver 1, 2 and 6 Before Linsolve = "EliminateDirichlet" "EliminateDirichlet“does not work in parallel - > comment or delete it in Solver 4 and 5Martina Schäfer May 2011 8

sif (2) in parallel it is impossible to use twice a solver with thesame name (here the case for FreeSurface in 4 and 5) get the FreeSurfaceSolver.f90 compile it to FreeSurfaceSolverBottom adjust the Solver 5 (Free Surface at the Bottom)Solver 5Equation = "Free Surface Bottom"Variable = String "Zs Bottom"Variable DOFs = 1Exported Variable 1 = String "Zs Bottom Residual"Exported Variable 1 DOFs = 1Procedure = "./FreeSurfaceSolverBottom""FreeSurfaceSolver"Martina Schäfer May 2011 9

Martina Schäfer May 2011 10

connect to vuori (hippu is not parallel)ssh –X vuori.csc.fi adjust your environmentload module elmer/latestcd $WRKDIR create file ELMERSOLVER_STARTINFO with thefollowing 2 linesteterousse.sif6 run it interactively for short tests:start the session: salloc -n 6 --mem-per-cpu=1000 -t 00:30:00 -p interactivestart the run: srun ElmerSolver_mpistop the session: exitsimulation (1)Martina Schäfer May 2011 11

for “real” simulations prepare a script for the batch queue(job scheduler software to share resources among users)script batchjob.sh sumbit a jobsbatch batchjob.sh check statussqueue –u USERNAMEsimulation (2)JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)440210 parallel CSC_job mschafer PD 0:00 1 (None)440213 parallel CSC_job mschafer R 1:53 1 (None)Martina Schäfer May 2011 12

#!/bin/csh###### parallel job script example##### name of your job#SBATCH -J CSC_jobsimulation (2)## system error message output file#SBATCH -e CSC_output_err_%j## system message output file#SBATCH -o CSC_output_%j## a per-process (soft) memory limit## limit is specified in MB## example: 1 GB is 1000#SBATCH --mem-per-cpu=2000Martina Schäfer May 2011 13

## how long a job takes, simulation wallclock time (2) hh:mm:ss#SBATCH -t 10:00:00##the number of processes (number of cores)#SBATCH -n 6## On Vuori there is 12 cores per node, so there could be 12mpi-processes per node## This just minimizes the number of nodes in your job.#SBATCH --ntasks-per-node=6##parallel queue#SBATCH -p parallel## run my MPI executablemodule load elmer/latestsrun ElmerSolver_mpiMartina Schäfer May 2011 14

simulation (3) check output (what was earlier printed on the screen)tail –f CSC_output_439011ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ParCommInit: Initialize #PEs: 6MAIN:MAIN: ==========================================MAIN: E L M E R S O L V E R S T A R T I N GMAIN: Library version: 6.2 (Rev: 5210)MAIN: Running in parallel using 6 tasks.MAIN: HYPRE library linked in.MAIN: MUMPS library linked in.MAIN: ==========================================…Martina Schäfer May 2011 15

simulation (4) put partitions together againElmerGrid 15 3 WithCavity/teterousse_step3 run took 7.6min (6processors) instead of88min (< 1 processor)Martina Schäfer May 2011 16