The Mare Nostrum Universe The biggest SPH simulation up to now

The Mare Nostrum Universe
Gustavo Yepes
Universidad Autónoma de Madrid

What is Mare Nostrum?
• IBM JS20 Blade Center:
– 4812 PPC processors
– 9.6 Tbytes of Memory
– Myrinet interconection.
– 233 Tbytes disk

What is Mare Nostrum?
• IBM JS20 Blade Center:
– 4812 PPC processors
– 9.6 Tbytes of Memory
– Myrinet interconection.
– 233 Tbytes disk
Still the most powerfull
supercomputer in Europe.
It is located at the BSC in
Barcelona

What is Mare Nostrum?
• IBM JS20 Blade Center:
– 4812 PPC processors
– 9.6 Tbytes of Memory
– Myrinet interconection.
– 233 Tbytes disk
Still the most powerfull
supercomputer in Europe.
It is located at the BSC in
Barcelona
A perfect place to create a Universe, not only because of its power…

The MareNostrum Universe
TREEPM+SPH simulation
ΛCDM model (WMAP1)
• 500/h Mpc 3 volume
• GADGET 2 code (Springel 2005)
• Adiabatic SPH+TREEPM Nbody
– 1024 3 FFT for the PM force.
– 15 kpc force resolution.
• 2x1024 3 dark and sph particles
– 10 9.33 partículas
– 8x10 9 M dark matter
– 10 9 M for gas particles
• 1 million dark halos bigger than
a typical galaxy (10 12 Mo)
• Simulation done at MareNostrum
– 512 processors (1/8 total power)
– 1Tbyte ram
– 500 wallclock hrs (29 cpu years)
– Output: 8600 Gbytes of data.
– Same computing power than the
Millenium Run.

Moore`s Law: Law
Capacity of professors
double every 18 months
N.Body simulations
double the number of
particles every 16.4
months
Extrapolating:
Extrapolating
10 10 partículas part culas in 2008
…but but it was done in
2004
Springel et al 2005
“Millenium Run”
“Hubbel Volume”

Moore`s Law: Law
Capacity of professors
double every 18 months
N.Body simulations
double the number of
particles every 16.4
months
Extrapolating:
Extrapolating
10 10 partículas part culas in 2008
…but but it was done in
2004
Springel et al 2005
“Millenium Run”
MareNostrum Universe (SPH)
“Hubbel Volume”

Moore`s Law: Law
Capacity of professors
double every 18 months
N.Body simulations
double the number of
particles every 16.4
months
Extrapolating:
Extrapolating
10 10 partículas part culas in 2008
…but but it was done in
2004
Springel et al 2005
Theoretical N-body body limit at MN
“Millenium Run”
MareNostrum Universe (SPH)
“Hubbel Volume”

Collaborators:
• S. Gottlober (AIP Germany)
• R. Sevilla (UAM, Spain)
• M. G. Rivero (UAM, Spain)
• M. Hoeft (Bremen, Germany)
• Massimo Meneghetti (ZAH, Heildelberg)
• Christian Wagner (AIP Germany)
• Arman Khalatyan (AIP, Germany)
• V. Turchaninov (IAM Moscow)
• A. Faltenbacher (USC, California)
• M. Plionis, S. Basilakos (NOA, Greece)
• F. Atrio (USAL, Spain)
• A. Doroshkevich (Moscow)
• …

Halo finder
• Hierarchical Friend of Friends analysis delivers
structures at virial overdensities and all
substructures (Klypin,Gottlober,Kravtsov 1997)
• Minimal Spanning Tree technique:
– Sorting particles in a cluster ordered sequence
• Mass, shapes, orientations, angular momentum,
particle list, are obtained from the MST for each
object.
• Full MPI implementation of the halo finder:
– Timing: Analysis of 1024 3 particlesis4 hrswith32
cpus.

Halo Mass function
• 4063 clusters with M > 10 14 h -1 Msun
• 58167 groups+clusters with M >10 13 h -1 Msun
• 506000 objects with M > 10 12 h -1 Msun
• More than 1 million objects with more than 100 particles

Spin parameter
Dark halos
λ = 0.023 for all overdensities
no hint that substructures have a different
spin distribution
“gas” halos
λ ~ 0.05

Shapes of halos
Shape of the dark matter distribution at
virial overdensity:
halos with more than 500 particles

Shape of dark matter halos at
Virial overdensity
Shapes of halos
Shape of dark matter halos at 64
times virial overdensity

Shape of dark matter halos at
Virial overdensity
Shape of halos
Shape of the corresponding gas
distribution in same halos

Dark Halos
Dark matter particles
Baryon oscillations

Dark Halos
Dark matter particles
Baryon oscillations

Dark Halos
Dark matter particles
Baryon oscillations

Baryon phase space diagram

Baryon phase space diagram

Baryon phase space diagram

HOT
Baryon phase space diagram
Warm-Hot
COLD

Evolution of baryon phases
(T > 10 7 K)
(10 5 K < T < 10 7 K)
(T < 10 5 K)

Evolution of baryon phases
All baryons WHIM baryons

Clusters of galaxies
Most massive cluster:

Baryon fraction in clusters

X-ray Temperature function

Comparison of Lx for two versions
of the same simulation:
Colored points: MareNostrum run
Black points: low-res version with
2x512 3 particles:
(Basilakos et al 2005
Ascasibar et al 2004-2005
Rubiño’s talk this meeting)
Scaling relations:
Resolution effects

Comparison of Lx for two versions
of the same simulation:
Colored points: MareNostrum run
Black points: low-res version with
2x512 3 particles:
(Basilakos et al 2005
Ascasibar et al 2004-2005
Rubiño’s talk this meeting)
Green Points are data..
Scaling relations:
Resolution effects

Evolution of scaling relations

Evolution of scaling relations

Evolution of scaling relations

Evolution of scaling relations

Evolution of scaling relations
(R. Sevilla’s Thesis)

Strong Lensing
• Arcs from our most massive cluster :
• Composite U,B,V using ray tracing technique.
• Collaboration with Massimo Meneghetti:
•Study role of cluster mergers on strong lensing efficiency.
•Create the largest database of lensed simulations from 4000
clusters between z=0 and 1.5.

SUMMARY
• The MareNostrum Universe simulation uses 2 billion particles to
simulate both the dark and baryonic matter. It is one of the largest
SPH simulations done up to now.
• It constitutes a very useful data base to do different studies of largescale
structure formation and distribution in both components.
• We have discussed here several of the analyses that are being
carried out:
– Halo identification and internal properties.
– Baryon oscillations in the LSS of halos.
– Baryon distribution.
– Clusters of galaxies: scaling relations, baryon fractions, lensing..
– S-Z signal from different structures (clusters, WHIM..)
– etc..
• We are open to any other interesting study you may have thought
– So please contact us if you’d like to put your hands on the tons of Gbytes of
data that the MareNostrum Universe is made of..

Thank you
–gustavo.yepes@uam.es
– sgottloeber@aip.de
• The MareNostrum supercomputer is run by
• We thank BSC for giving access to this facility
• http://www.bsc.es