Parallel version - CSC
Parallel version - CSC
Parallel version - CSC
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
Tête Rousse in <strong>Parallel</strong>Schäfer May 2011 1Martina Schäferfer, , Arctic Centre, UoL
steps parallel mesh changes to the sif run the simulation Martina Schäfer May 2011 2
Martina Schäfer May 2011 3
create mesh in 6 partitionsmesh (1)ElmerGrid 2 2 WithCavity –halo –metis 6 4elmer input formatmesh folder# procMartina Schäfer May 2011 4
mesh (2) after the simulation to connect the single .ep filestogetherElmerGrid 15 3 WithCavity/teterousse_step3Martina Schäfer May 2011 5
mesh (3)Martina Schäfer May 2011 6
Martina Schäfer May 2011 7
sif (1) direct method umfpack does not work in parallel use mumps instead replace Linear System Direct Method = umfpackwith Linear System Direct Method = mumps in Solver 1, 2 and 6 Before Linsolve = "EliminateDirichlet" "EliminateDirichlet“does not work in parallel - > comment or delete it in Solver 4 and 5Martina Schäfer May 2011 8
sif (2) in parallel it is impossible to use twice a solver with thesame name (here the case for FreeSurface in 4 and 5) get the FreeSurfaceSolver.f90 compile it to FreeSurfaceSolverBottom adjust the Solver 5 (Free Surface at the Bottom)Solver 5Equation = "Free Surface Bottom"Variable = String "Zs Bottom"Variable DOFs = 1Exported Variable 1 = String "Zs Bottom Residual"Exported Variable 1 DOFs = 1Procedure = "./FreeSurfaceSolverBottom""FreeSurfaceSolver"Martina Schäfer May 2011 9
Martina Schäfer May 2011 10
connect to vuori (hippu is not parallel)ssh –X vuori.csc.fi adjust your environmentload module elmer/latestcd $WRKDIR create file ELMERSOLVER_STARTINFO with thefollowing 2 linesteterousse.sif6 run it interactively for short tests:start the session: salloc -n 6 --mem-per-cpu=1000 -t 00:30:00 -p interactivestart the run: srun ElmerSolver_mpistop the session: exitsimulation (1)Martina Schäfer May 2011 11
for “real” simulations prepare a script for the batch queue(job scheduler software to share resources among users)script batchjob.sh sumbit a jobsbatch batchjob.sh check statussqueue –u USERNAMEsimulation (2)JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)440210 parallel <strong>CSC</strong>_job mschafer PD 0:00 1 (None)440213 parallel <strong>CSC</strong>_job mschafer R 1:53 1 (None)Martina Schäfer May 2011 12
#!/bin/csh###### parallel job script example##### name of your job#SBATCH -J <strong>CSC</strong>_jobsimulation (2)## system error message output file#SBATCH -e <strong>CSC</strong>_output_err_%j## system message output file#SBATCH -o <strong>CSC</strong>_output_%j## a per-process (soft) memory limit## limit is specified in MB## example: 1 GB is 1000#SBATCH --mem-per-cpu=2000Martina Schäfer May 2011 13
## how long a job takes, simulation wallclock time (2) hh:mm:ss#SBATCH -t 10:00:00##the number of processes (number of cores)#SBATCH -n 6## On Vuori there is 12 cores per node, so there could be 12mpi-processes per node## This just minimizes the number of nodes in your job.#SBATCH --ntasks-per-node=6##parallel queue#SBATCH -p parallel## run my MPI executablemodule load elmer/latestsrun ElmerSolver_mpiMartina Schäfer May 2011 14
simulation (3) check output (what was earlier printed on the screen)tail –f <strong>CSC</strong>_output_439011ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ELMER SOLVER (v 6.2) STARTED AT: 2011/06/01 15:55:24ParCommInit: Initialize #PEs: 6MAIN:MAIN: ==========================================MAIN: E L M E R S O L V E R S T A R T I N GMAIN: Library <strong>version</strong>: 6.2 (Rev: 5210)MAIN: Running in parallel using 6 tasks.MAIN: HYPRE library linked in.MAIN: MUMPS library linked in.MAIN: ==========================================…Martina Schäfer May 2011 15
simulation (4) put partitions together againElmerGrid 15 3 WithCavity/teterousse_step3 run took 7.6min (6processors) instead of88min (< 1 processor)Martina Schäfer May 2011 16