Tree Poisson solver

Tree-based self-gravity solver 

R. Wünsch I. Berentzen 

A. P. Whitworth R. Banerjee 

Features: 

• two versions for Flash2 and Flash3 

• Barnes & Hut octal tree with monopole moments 

• works with 3D Cartezian coords, AMR tree needed 

• isolated and periodic boundaries 

◮ periodic: Ewald method 

• efficient MPI communication 

• interaction lists 

◮ nearby cells and tree nodes are not tested for opening angle criterion 

• ported to GPUs (by Ingo Berentzen) 

Richard Wünsch, Flash workshop at the Hamburg Observatory, 15th February 2012 1/26

Algorithm overview 

• Parameters: 

◮ tree limangle (θ lim ) . . . REAL . . . 1.0 - 0.5 

◮ tree ilist . . . INTEGER . . . 0 or 1 

• Algorithm: (as in Grid solvePoisson()) 

if (grid changed .eq. 1) then 

call treeComBlkProperties() (nodetype, lrene, child, neigh, coords) 1% 

if (tree ilist .eq. 1) call treeFindNeighbours() 0% 

endif 

call gr treeBuildTree() 3% 

call gr treeExchangeTrees() 3% 

call gr treePotential(idensvar, ipotvar) 93% 

call gr treeDestroyTree() 0% 

◮ routines that include MPI communication marked red 

◮ relative times for a collapse of the BE sphere 

(512 CPUs, θ lim = 0.5, 76000 leaf blocks) 


Communication of grid properties 

• treeComBlkProperties() 

◮ nodetype, lrene, child, neigh and cell coords in each block 

◮ memory: MAXBLOCKS × nCPUs × (16×INT + 27×REAL) 

◮ for MAXBLOCKS = 1000, nCPU = 1000 → 300 MB 

◮ only active values communicated 

• treeFindNeighbours() (needed by interaction lists) 

◮ nds 26 neighbours in all (incl. diagonal) directions 

◮ tr surbox(2, 27, MAXBLOCKS) 

→ records block number and cpu of each neighbour 


Build Tree 

1. build trees in blocks 

◮ octal tree with log 2 (nxb) levels 

2. communicate mass and mass 

centre pos. of all leaf blocks 

3. build ParentTree on each CPU 

◮ ParentTree(4, MAXBLOCKS, nCPU) 

3 

0 

2 

1 


• for 8 × 8 × 8 blocks: 

Block tree in RAM 

level 0 

level 1 

m x mc y mc z mc 

1 2 3 4 

5 9 13 17 21 25 29 33 

level 2 

36 68 100 132 164 196 228 260 

level 3 

masses only (mc given by cell coordinates), 8 3 

= 512 

292 804 

Tree size = 8 L +4 

L−1 

∑ 

i=0 

8 i = 8 L +4 8L − 1 

7 

L . . . number of the lowest level (e.g. 3 for 8 × 8 × 8 blocks) 

tree nodes identified by multi-index - integer array of size L: (l 1 , l 2 , l 3 ); l i = 

◮ 1-8 . . . number of node on i-th level 

◮ 0 . . . multi-index (i.e. node) is of level i-1 


Communication of block trees 

1. determine tree levels to be sent 

◮ distance between a given block and all 

blocks on a given CPU 

2. communicate tree levels 

◮ all values for a given CPU packed into 

a single message 

3. allocate space for block trees 

from other CPUs 

◮ dynamic memory allocation to avoid 

wasting of memory 

4. communicate block trees 

◮ all block trees for a given CPU packed 

into a single message 

CPU 1 

level 3 

level 2 

level 1 

level 0 

CPU 0 


• 5 types of iteractions: 

1. cell – cell 

2. cell – block tree node 

◮ θ lim checked for each cell 

3. cell – block 

Tree walk 

◮ θ lim checked once for the whole block 

4. cell – cell (with interaction lists) 

◮ distance pre-calculated 

5. cell – block tree node (with interaction lists) 

◮ θ lim criterion pre-calculated 

[ 02-13-2012 15:15:36.682 ] [TREE]: cell-cell distances: 0.548E+08, per zone: 111.4 

[ 02-13-2012 15:15:36.692 ] [TREE]: cell-node distances: 0.756E+09, per zone: 1537.1 

[ 02-13-2012 15:15:36.733 ] [TREE]: cell-block distances: 0.169E+08, per zone: 34.3 

[ 02-13-2012 15:15:36.773 ] [TREE]: IL cell-cell distances: 0.169E+09, per zone: 343.4 

[ 02-13-2012 15:15:36.803 ] [TREE]: IL cell-node distances: 0.461E+09, per zone: 938.4 


Interaction lists 

• relative positions of cells and tree nodes are known 

• for each cell can be found: 

◮ list of cells with which it interacts 

◮ list of block tree nodes with which it interacts 

• lists do not depend on lrefine 

• for cells/nodes within a block and 26 surrounding blocks 

• makes tree walk faster by ∼ 25% 

◮ more ecient for smaller sims 

• costs memory: 

◮ 8×8×8 blocks: 

∼ 200 MB 

◮ 16×16×16 blocks: ∼ 700 MB 


Test: Bonnor-Ebert sphere 

• mass: M = 1 M ⊙ 

• temperature: T = 10 K (BES), T amb = 10 4 K (ambient) 

• unstable: ξ 0 = 10 (threshold value is 6.5) 

• radius: R = 0.041 pc 

• accuracy tests: 

◮ "uniform grid": lrene min = lrene max = 5 

→ 4096 leaf blocks 

◮ "AMR grid": lrene min = 1, lrene max = 5 

→ 1240 leaf blocks 

→ renement controlled by Jeans length 

• performance tests: 

◮ "AMR grid": lrene min = 1, lrene max = 8 

→ 76000 leaf blocks (run on 64 − 512 CPUs) 

◮ integrated for 10 time-steps 


Flash 3: error in Φ 


Flash 3: error in Φ (lref min = lref max = 5) 

log 10 [(Φ-Φ anl )/|Φ anl (0)|] 

0.025 

0.02 

0.015 

0.01 

0.005 

0 

Tree, θ lim = 1.0 



Multigrid, mpole_lmax = 0 



-0.005 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 3: error in Φ (lref min = lref max = 5) 

log 10 [(Φ-Φ anl )/|Φ anl (0)|] 

0.025 

0.02 

0.015 

0.01 

0.005 

0 








-0.005 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 3: error in F r (lref min = lref max = 5) 

log 10 [(F r -F r,anl )/|F r,anl |] 

0.1 

0.05 

0 

-0.05 







-0.1 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 3: error in F r (lref min = lref max = 5) 


0.1 

0.05 

0 

-0.05 








-0.1 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 3: error in F r (lref min = 1, lref max = 6) 


0.1 

0.05 

0 

-0.05 







-0.1 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 2: error in Φ (lref min = 1, lref max = 6) 

log 10 [(Φ-Φ anl )/|Φ anl (0)|] 

0.025 

0.02 

0.015 

0.01 

0.005 

0 







-0.005 

0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 2: error in F r (lref min = 1, lref max = 6) 

0.3 

0.2 


0.1 

0 

-0.1 

-0.2 

-0.3 







0 0.01 0.02 0.03 0.04 0.05 

r [pc] 


Flash 3, BES: time(nCPUs) 

350 

300 

Seconds per timestep 

250 

200 

150 

100 

50 

other 

com/gcell 

tree walk/fft 

hydro 

0 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

64 128 256 512 

Number of CPUs 


Flash 3, BES: relative time 

1 

other 

com/gcell 

tree walk/fft 

hydro 

Seconds per timestep 

0.8 

0.6 

0.4 

0.2 

0 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

64 128 256 512 



Flash 2, test: expanding shell 

M shell = 2 × 10 4 M ⊙ 

-20 

T shell = 10 K 

-22 

R shell,0 = 10 pc 

-24 

V shell,0 = 2.2 km s −1 

-26 

R shell,max = 23 pc 

-28 

P ext = 10 −17 , 10 −13 

-30 

or 5 × 10 −13 dyne cm −2 -32 

log(ρ) [g/cm 3 ], (log(T) - 32) [K] 

ρ ∝ sech 2 

T = 10 4 K 

2.5 

1.5 

0.5 

T = 10 K 

0 5 10 15 20 25 30 0 

r [pc] 

log ρ 

log(T) - 32 

v 

2 

1 

v [km/s] 


Flash 2, shell: time(nCPUs) 

70,000 

60,000 

Seconds per evolution 

50,000 

40,000 

30,000 

20,000 

10,000 

other 

com/gcell 

tree walk/fft 

hydro 

0 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

tree_1.0 

tree_0.5 

mg 

64 128 256 



Flash 3, BES: speedup 

speedup (64 x t 64 /t) 

700 

600 

500 

400 

300 

200 

100 

90 

80 

70 

60 

50 

hydro 

multigrid 

tree, θ lim = 0.5 


64 128 256 512 



Flash 3: time(number of blocks) 

seconds per time-step 

1000 

100 

10 

1 

0.1 

0.01 

64 CPUs, lrefine_min = lrefine_max, ilist=1 

hydro 


N log(N) 

N 

0.001 

64 512 4096 32768 

Number of blocks 


Flash 3, BES: time(θ lim ) 

seconds per time-step 

10000 

1000 

100 

10 

1 

64 CPUs, lrefine_min = lrefine_max = 5, ilist=1 

θ lim 

-2 

θ lim 

-3 

0.1 

0.2 0.5 1 

θ lim 


GPU version 

• tree walk ported to GPUs by Ingo Berentzen 

• 5-10 faster, comparable to hydro 

Future 

• elimination of MAXBLOCKS×nCPU size arrays 

• uniform grid 

• quadrupole (higher) moments 


Download at: 

http://galaxy.asu.cas.cz/˜richard/tree-solver/

Download at: 

http://galaxy.asu.cas.cz/˜richard/tree-solver/ 

Thank you!

Tree Poisson solver

Create successful ePaper yourself

Delete template?

Save as template?