HPCToolkit: Performance Analysis on Heterogeneous ...

<strong>HPCToolkit</strong>: 

A Tool for <strong>Performance</strong> <strong>Analysis</strong> 

on Heterogeneous Supercomputers 

Milind Chabbi, Karthik Murthy, 

Mike Fagan, and John Mellor-Crummey 

Rice University 

GTC 2013 

San Jose 

March 21, 2013

Architectural Shift in Supercomputing 

18K 16-core 18K Tesla K20 

Milind Chabbi 

<strong>HPCToolkit</strong>: A Tool for <strong>Performance</strong> <strong>Analysis</strong> on Heterogeneous Supercomputers

Architectural Shift in Supercomputing 

Keeneland (KFS ) at University of Tennessee 

★ 

264 nodes of two 8-core Xeon E5 CPUs + 3 Nvidia M2090 GPUs 

2, 8-core 

Xeon E5 

3 Tesla M2090 

Milind Chabbi 


Challenge: Hybrid Applications 

Migrating scientific applications to heterogeneous systems 

✦ 

Molecular dynamics 

★ 

★ 

LAMMPS 

NAMD 

These codes have a 

highly-tuned CPU part 

✦ 

Hydrodynamics 

and an emerging GPU part 

★ 

LULESH 

<strong>Performance</strong> analysis tools 

✦ 

Lattice field theory 

play a vital role in 

★ 

Chroma 

tuning these applications 

Milind Chabbi 


Project Goals 

• Provide performance analysis tools for emerging 

heterogeneous supercomputers 

✦ 

GPU kernels alone → whole application performance analysis 

• Provide performance improvement “insights” 

✦ 

Opportunities for tuning, rather than resource consumption 

• Desired properties of the tool 

✦ 

✦ 

✦ 

Introduce minimal execution perturbation 

Collect compact measurement data at scale 

Provide language / programming model independence 

Milind Chabbi 


Accelerated LAMMPS 

Molecular dynamics code LAMMPS has two accelerated 

versions: 

LAMMPS-CUDA 

LAMMPS-GPU 

Offloads many timesteps of force 

calculation GPUs 

3 GPUs busy; 3 CPU cores 

idle and 13 CPU cores 

unused 

Atom data moves in and out of 

GPU each timestep 

CPU and GPU computations 

are not overlapped 

Emerging support for CPU-GPU 

load balancing 

Milind Chabbi 


LAMMPS-CUDA on Keeneland 

CPU 

Mostly Idle 

GPU STREAM 

GPU STREAM 

Time 

Milind Chabbi 


Idleness Wastes Compute Power 

• Offloading entire computation to GPUs wastes CPU 

• 

Performing entire computation on CPUs wastes GPU 

Better approach: 

• Overlap CPU and GPU computation 

✦ 

✦ 

Divide principal computation between CPU and GPU 

Use CPU to prepare next work while GPU is busy 

Milind Chabbi 


Focus on Resource (under) Utilization 

• To achieve high performance on heterogeneous systems, 

applications should exploit all compute resources 

• Problem: “hotspot analysis” of today’s performance 

analysis tools is insufficient for tuning 

✦ 

✦ 

✦ 

Focuses on “most consumed” resources 

Identifies only symptoms of performance problems 

Does not identify causes of problems 

Milind Chabbi 


Our Methodology: Pinpoint and Quantify Idleness 

• Quantify resource idleness 

• Attribute idleness to its causes 

✦ 

✦ 

If GPU is idle → blame CPU code for not offloading (enough) work 

If CPU is waiting for results from GPU → blame GPU kernel(s) 

involved 

Milind Chabbi 


Our Methodology: Pinpoint and Quantify Idleness 

• Quantify resource idleness 

• Attribute idleness to its causes 

✦ 

✦ 

If GPU is idle → blame CPU code for not offloading (enough) work 

If CPU is waiting for results from GPU → blame GPU kernel(s) 

involved 

a.k.a. “CPU-GPU Blame Shifting” 

Milind Chabbi 


Quantifying Tuning Opportunities with 

CPU-GPU Blame Shifting - 1 

5% Idle 40% Idle 

CPU 

WORK 

SYNC 

CPU 

WORK 

SYNC 

Kernel A 

Kernel B 

Time 

5% expected gain by 

tuning Kernel A 

40% expected gain 

by tuning Kernel B 

Top GPU-kernel may not be the best candidate for tuning 

Milind Chabbi 




5% Idle 40% Idle 

CPU 

WORK 

SYNC 

CPU 

WORK 

SYNC 

Kernel A 

Kernel B 

Time 

5% expected gain by 

tuning Kernel A 

40% expected gain 

by tuning Kernel B 

Hotspot analysis 

Blame shifting 

Top GPU-kernel may not be the best candidate for tuning 

Milind Chabbi 


Hotspot vs. Blame Shifting on LULESH 

CalcFBHourglassForceForElems 1 

CalcKinematicsForElems 2 

CalcMonotonicQGradientsForElems 3 

CalcHourglassControlForElems 4 

IntegrateStressForElems 5 

AddNodeForcesFromElems2 6 

AddNodeForcesFromElems 7 

CalcPressureForElems 8 

CalcMonotonicQRegionForElems 9 

EvalEOSForElemsPart1 10 

CalcEnergyForElemsPart2 11 


CalcPositionForNodes 13 

CalcVelocityForNodes 14 


CalcAccelerationForNodes 16 

CalcLagrangeElementsPart2 17 


CalcSoundSpeedForElems 19 


InitStressTermsForElems 21 

CalcCourantConstraintForElems 22 

CalcHydroConstraintForElems 23 

ApplyMaterialPropertiesForElemsPart1 24 

UpdateVolumesForElems 25 

ApplyAccelerationBoundaryConditionsForNo 26 

des 

Milind Chabbi 

Hotspot 

1 CalcFBHourglassForceForElems 

2 CalcHourglassControlForElems 

3 IntegrateStressForElems 

4 AddNodeForcesFromElems2 

5 AddNodeForcesFromElems 

6 CalcKinematicsForElems 

7 EvalEOSForElemsPart2 

8 CalcSoundSpeedForElems 

9 CalcLagrangeElementsPart2 

10 InitStressTermsForElems 

11 CalcCourantConstraintForElems 

12 CalcHydroConstraintForElems 

13 UpdateVolumesForElems 

14 CalcMonotonicQGradientsForElems 


16 CalcMonotonicQRegionForElems 

17 CalcPressureForElems 

18 CalcEnergyForElemsPart2 



21 ApplyMaterialPropertiesForElemsPart1 


23 CalcPositionForNodes 

24 CalcVelocityForNodes 

25 CalcAccelerationForNodes 

26 ApplyAccelerationBoundaryConditionsForNo 

des 

Blame Shifting 





























des 

Milind Chabbi 

Hotspot 



























des 






























des 

Milind Chabbi 

Hotspot 



























des 






























des 

Milind Chabbi 

Hotspot 



























des 






























des 

Milind Chabbi 

Hotspot 



























des 






























des 

Milind Chabbi 

Hotspot 



























des 





Vice versa is also true 

CPU PART A 

S 

Y 

N 

C 

CPU PART B 

S 

Y 

N 

C 

KernelA 

GPU IDLE 

KernelB 

Time 

Tuning “CPU PART A” reduces critical path 

Milind Chabbi 


<strong>HPCToolkit</strong> 

<strong>Performance</strong> tool for large parallel systems 

• Supports multilingual, fully-optimized, statically or 

dynamically linked applications (no source modification) 

• Measures performance using asynchronous sampling of 

timers and hardware performance counters 

✦ 

Low overhead (under 5%) for both profiling and tracing 

• Attributes performance to full call paths 

✦ 

Pthread, OpenMP, MPI, and any combination 

• Decentralized → Scales to 1000s of nodes 

• Provides GUIs for code- and time- centric analysis 

Milind Chabbi 


Enhancing <strong>HPCToolkit</strong> for 

Heterogeneous Supercomputers 

• Problem: GPUs (natively) don’t support sampling 

• Approach: Instrument GPU runtime APIs to monitor 

asynchronous activities via cudaEvents; use CPU events 

as sampling points to interrogate GPU 

• CPU-GPU blame shifting 

✦ 

✦ 

Attributes blame to causes rather than symptoms on the fly 

Delivers deep insight without recording activity traces 

• Support for full calling context on GPUs to distinguish 

same kernels launched from different calling contexts 

Milind Chabbi 


CPU-GPU Blame Shifting in Action in 


CPU 

hread 

GPU 

stream 

Time 

Milind Chabbi 




CPU 

K 

e 

r 

n 

e 

l 

hread 

GPU 

stream 

Time 

Milind Chabbi 




CPU 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

hread 

GPU 

stream 

Time 

Milind Chabbi 




CPU 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

Overlap 

cudaDevice 

Synchronize() 

hread 

GPU 

e 

v 

e 

n 

t 

Kernel Execution 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

cudaDevice 


GPU 

e 

v 

e 

n 

t 


stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


GPU 

e 

v 

e 

n 

t 


stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


EventQuery 

EventQuery 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

Blame 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

Blame 

e 

v 

e 

n 

t 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

CPU ONLY 

GPU idle 

GPU idle 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

Blame 

e 

v 

e 

n 

t 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

CPU ONLY 

GPU idle 

GPU idle 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

Blame 

e 

v 

e 

n 

t 

GPU IDLE 

stream 

60% 

Time 

40% 

Milind Chabbi 




CPU 

hread 

e 

v 

e 

n 

t 

K 

e 

r 

n 

e 

l 

e 

v 

e 

n 

t 

EventQuery 

Overlap 

EventQuery 

cudaDevice 


CPU IDLE 

EventQuery 

EventQuery 

CPU ONLY 

GPU idle 

Blame 

GPU idle 

GPU 

e 

v 

e 

n 

t 


GPU ONLY 

Blame 

e 

v 

e 

n 

t 

GPU IDLE 

stream 

60% 

Time 

40% 

Milind Chabbi 


Advanced Features 

• <strong>HPCToolkit</strong>’s CPU-GPU blame shifting supports 

✦ 

✦ 

✦ 

✦ 

Multiple GPU streams 

Multiple threads in a process 

Multiple MPI ranks sharing same GPU 

Any combination of above 

• GPU hardware counters for kernel-level hotspot analysis 

• <strong>Performance</strong> modeling when multiplexing GPUs 

Milind Chabbi 


Insights via Blame Shifting in 

LULESH CUDA 

• LULESH: Shock Hydrodynamics code 

✦ 

✦ 

One of five challenge problems in the DARPA UHPC program 

Compute intensive simulation of complex multi-material motion 

• CUDA version available from LLNL 

Milind Chabbi 


Milind Chabbi 


Insights via Blame Shifting in 

LULESH CUDA 

CPU 

CPU IDLE 

CudaMalloc() 

CudaFree() 

GPU IDLE 

GPU IDLE 

GPU 

Time 

Milind Chabbi 


LULESH CUDA Memory Allocation 

9.E+07 

8.E+07 

Total bytes allocated 

7.E+07 

6.E+07 

5.E+07 

4.E+07 

3.E+07 

2.E+07 

1.E+07 

0.E+00 

Time step 1 Time step 2 Time step 3 Time step 4 

1 

11 

21 

31 

41 

51 

61 

71 

81 

91 

101 

111 

121 

131 

141 

151 

161 

171 

181 

191 

201 

211 

221 

231 

241 

251 

261 

271 

281 

291 

301 

311 

321 

Memory request with time 

Milind Chabbi 


LULESH CUDA Memory Allocation 

9.E+07 

8.E+07 

Total bytes allocated 

7.E+07 

6.E+07 

5.E+07 

4.E+07 

3.E+07 

2.E+07 

1.E+07 

0.E+00 

Time step 1 Time step 2 Time step 3 Time step 4 

1 

11 

21 

31 

41 

51 

61 

71 

81 

91 

101 

111 

121 

131 

141 

151 

161 

171 

181 

191 

201 

211 

221 

231 

241 

251 

261 

271 

281 

291 

301 

311 

321 

Memory request with time 

Hoisting repeated allocation/free improved GPU utilization from 

65% to 95% and reduced execution time by 30% 

Milind Chabbi 


Understanding Temporal Behavior 

• Profiling compresses out the temporal dimension 

✦ 

Temporal behavior e.g., serialization are invisible in profiles 

• What can we do Trace call path samples 

✦ 

✦ 

✦ 

On each sample trace call path sample for each CPU and GPU 

Organize samples along a timeline 

Assign each procedure a color; view depth slice of an execution 

Compute 

resources 

Orders of 

magnitude smaller 

trace sizes 

Milind Chabbi 


HPCTraceviewer to Identify 

<strong>Performance</strong> Issue on 

Keeneland Cluster 

DEMO 

Milind Chabbi 


Conclusions 

<strong>HPCToolkit</strong> is a performance analysis tool for 

heterogeneous systems 

✦ 

Sampling-centric approach 

★ 

Low overhead and compact measurement data 

✦ 

CPU-GPU blame shifting 

★ 

Pinpoints and quantifies code fragments (CPU and GPU) 

worth tuning 

✦ 

Open source, BSD license 

hpctoolkit.org 

Milind Chabbi 


Conclusions 

<strong>HPCToolkit</strong> is a performance analysis tool for 

heterogeneous systems 

✦ 

Sampling-centric approach 

★ 

Low overhead and compact measurement data 

✦ 

CPU-GPU blame shifting 

★ 

Pinpoints and quantifies code fragments (CPU and GPU) 

worth tuning 

✦ 

Open source, BSD license 

hpctoolkit.org 

Milind Chabbi

HPCToolkit: Performance Analysis on Heterogeneous ...

Create successful ePaper yourself

Delete template?

Save as template?