Jim Tomkins

The Red Storm Story
Red Storm Update
Jim Tomkins
Jim Tomkins
SOS 11
Key West, Florida
June 12 - 14, 2007
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.
Sandia is a multiprogram laboratory operated by Sandia Corporation, a Lockheed Martin Company,
for the United States Department of Energy’s National Nuclear Security Administration
under contract DE-AC04-94AL85000.

Upgrade Changes
• SeaStar 2.1 NIC/Router Chips
– 2 x Injection Bandwidth
• 5th Row (25% more compute and service nodes)
• Dual Core Opterons
– 2.5 x More Compute Processors
– 20% Higher Clock Rate
• New OS’s (Linux 2.6, CVN)
– Virtual node model –– 1PE or 2PE / node
• New File Systems
– Lustre 1.4 on Linux 2.6
• Complete: mid-November, 2006

Normally
Classified
I/O and
Service Nodes
Upgraded Red Storm Layout
(27 x 20 x 24 Compute Node Mesh)
Switchable Nodes
Disconnect Cabinets
Disk storage system
not shown
Normally
Unclassified
I/O and
Service Nodes

Upgraded Red Storm

Theoretical Peak Performance
(Compute Nodes Only)
Red Storm Comparison
Red Storm
(initial operations)
41.47 TF 124.42 TF
HPL Performance 36.19 TF 101.400 TF
Red Storm
(post-upgrade)
Compute Nodes / Processors 10368 / 10368 12960 / 25920
Service and I/O Nodes /
Processors
256 + 256
256 + 256
320 + 320
640 + 640
Processor 2.0 GHz Opteron 2.4 GHz Dual Core
Opteron
Total Memory (TB) 33.38 39.19
System Memory B/W (TB/s) 55.2 82.9
Topology 27 x 16 x 24 27 x 20 x 24
Bi-Section B/W (TB/s) 3.7, 6.2, 8.3 4.6, 6.2, 10.4
Power / Cooling (MW) 1.7 2.5
System Size ~3100 ft 2 ~3800 ft 2

Some Performance Data

#2 on Top 500 list at 101.4 TF

Single Core Versus Dual Core

Single Core Versus Dual Core

Single Core Versus Dual Core
1.20
1.00
0.80
0.60
0.40
0.20
0.00
CTH (2000)
SAGE (2048)
UMT2K (3200)
ITS (3200)
Red Storm (SN vs. VN)
SN = 1PE/socket, VN = 2PE/socket
PRTSN (4096)
SAGE (4500)
UMT2K (4500)
ITS (4500)
sPPM (4500)
Application (PEs)
CTH (6480)
PRTSN (6480)
sPPM (6561)
PRTSN (8930)
SN (post-upgrade)
VN (post-upgrade)
CTH (9000)
sPPM (9000)

John Daly’s Production Work
• Recent results have shown that for John’s application we
are getting a large fraction of the second core.
– John’s application code runs on 3000 dual core nodes in
the same time as 5000 single core nodes. This means
that he is getting about 70% efficiency out of using the
second core.

Red Storm Upgraded to ~500TF
• Upgrade Existing System to Quad Core XT4 Specifications
– Replace all Compute Boards
– 2.4 GHz AMD Opteron Quad Core Processors
– New DDR2 Memory (>75 TB)
– Replace Power Supplies
– Replace Cabinet Cooling Fans
– Reuse Seastar Mezzanines, L0s, Cabinets, Cables, PDUs, etc.
• A Few System Parameters
– ~500 TF
– >75 TB of Memory
– >1 PB of Disk Storage
• Timeframe - Early 2008

Red Storm Impact
• Cray and Sandia with the support of NNSA/ASC
have created one of the most successful new
supercomputers ever.
• There are now 17 sites with Red Storm / XT3 / XT4
computer systems.

ARMY/HPC
AWE (England)
CSC (Finland)
CSCS
(Switzerland)
DOD/ERDC
EPSRC (UK)
JAIST (Japan)
NERSC
ORNL
PSC
SNL
Red Storm / XT3 / XT4 Sites
SS-634 (non-US)
SS-635
SS-643
SS-661
U of Tokyo (JST)
U of Western Australia
Worldwide
Count of Sites
(Cumulative)
20
15
10
Number of XT3/XT4 Sites Worldwide
5
0
1H05 1 2H05 2 1H06 3 2H06 4 1H07 5 2H07 6
TFLOPS
800
600
400
200
0
Half Years
TFLOPS available Worldwide
1 2 3 4 5 6
1H05 2H05 1H06 2H06 1H07 2H07
Half Years