Advanced Computer Architecture Exercise Interconnection Networks

Advanced Computer Architecture
Exercise
Prof. Dr.-Ing. Axel Hunger
Department of Electrical Engineering and Information
Technology
Institute for Computer Engineering.
Dipl.-Ing. Kevin Moheb
Institute for Computer Engineering
BB 321, Tel: 0203 – 379 - 3200
E-mail: kevin.moheb@uni-due.de
Interconnection Networks
27.06.2008
1

Class of networks scaling with N
Logical Properties:
◦ distance, degree
Physical properties
◦ length, width
VLSI technology determines switch degree
Fully Connected Network:
• Every node is connected to all other nodes using N- 1 direct links
• N(N-1)/2 Links -> O(N2 ) complexity
• NNode d DDegree: N -1 1
• Diameter = 1
• Average Distance = 1
• Bisection Width = (N/2) 2
Chain (Linear Array):
◦ N-1 Links -> O(N) complexity
◦ Node Degree: 1-2
◦ Diameter = N -1
◦ Average Distance = (N+1)/3 ~ N/3
◦ Bisection Width = 1
Ring (1-D Torus):
◦ N bi-directional Links -> O(N) complexity
◦ Node Degree: 2
◦ Diameter
N is even: N/2
N is s odd: odd (N-1)/2 ( )/
◦ Average Distance:
N is even: N2 /4/(N-1)
N is odd: (N+1)/4
~ N/4
◦ Bisection Width = 2
3
4
27.06.2008
2

2D Mesh and Torus
• Mesh:
– 2k(k-1) links
– Node degree: 2-4
– Diameter: 2(k-1)
– Average distance?
– Bisection width: k
• Torus:
– 2k2links – NNode d ddegree: 4
– Diameter: k - 1
– Average distance?
– Bisection width: 2k
1D mesh
d-dimensional mesh or torus :
– N = kd-1 x kd-2 ...x k1 x k0 nodes
– Described by d-vector of coordinates (id-1, ..., i0) – Where 0 < ij ≤ kj for 0 ≤ j ≤ d-1
d-dimensional k-ary mesh: N = kd – k = d 1D torus 2D mesh 2D torus 3D mesh
√N
– Described by d-vector of radix k coordinate.
– Diameter = d·(k -1)
d-dimensional k-ary torus:
– Edges wrap around, every node has degree 2d and is
connected to nodes that differ by one (mod k) in every
dimension.
– d-dimensional k-ary cube: uni-directional links
5
6
27.06.2008
3

Distance:
◦ Relative distance: R = (b d-1 -a d-1, ... , b 0 -a 0 )
◦ Traverse r i = b i -a i hops in each dimension
i i i p
Average Distance:
◦ d x k/3 for mesh
◦ d x k/4 for torus
Degree:
◦ d to 2d for mesh
◦ 2d for torus
Bisection width:
◦ k d-1 for mesh
◦ 2k d-1 for torus
1824 node Paragon: 16 x 114 array
7
8
27.06.2008
4

Also called binary n-cubes
Dimension = n = log2N Number of nodes =N=2n Number of nodes = N = 2
Diameter: O(log2N) hops
Good bisection width: N/2
Number of bidirectional links: N·(log2N)/2 Node degree is n = log2N 0-D 1-D 2-D 3-D
4-D
0010
0110
0100
0011
0000 0001
0111
0101
1010
1110
1011
1100
1000 1001
1111
1101
5-D
9
10
27.06.2008
5

Trees
• Diameter and average distance are logarithmic.
– kk-ary ary tree tree, height d =log = logk N
• Fixed degree k.
• Low bisection width = 1
Fat Tree
Fatter higher bandwidth links (more connections in reality)
Bisection width scales with number of nodes N.
11
12
27.06.2008
6

Switch
0
1
Either connects the same set
of processors or connects
processors with memory.
Complexity: N/2 x log 2 N
Exactly one route from any
source to any destination node
Same length for all routes
Bisection: N/2
Diameter: log 2 N
Bus is shared by processors and memories.
Bus provides uniform access to shared memory (UMA).
When Wh bus b saturates, performance f of f system degrades. d d
For this reason, bus-based systems do not scale to
more than 30-40 processors.
memory … memory
bus
processor processor
…
13
14
27.06.2008
7

Crossbar switch connects m processors and n memories
with distinct paths between each processor/memory pair
Crossbar provides uniform access to shared memory (UMA)
m·n switches required for m processors and n memories
Crossbar scalable in terms of performance but not in terms
of cost
P1
P2
P3
P4
P5
M1
M2 M3 M4 M5
Topology Degree Diameter Ave Dist Bisection Ave Dist N=1024
1D Array 2 N-1 ~ N / 3 1 342
1D Ring 2 N/2 (even) ~ N/4 2 256
2D Mesh 2-4 2 (N1/2 -1) 2/3 N1/2 N1/2 21
2D Torus 4 N1/2 1/2 N1/2 2N1/2 16
Hypercube n=log2 N n ~ n/2 N/2 5
All have some bad permutations
◦ Many popular permutations are very bad for meshes
(transpose)
◦ Randomness in wiring or routing makes it hard to find a
bad one!
15
16
27.06.2008
8

Wide links, smaller routing delay
Tremendous variation
d = 2 or d = 3
◦ Short wires, , easy y to build
◦ Many hops, low bisection bandwidth
◦ Requires traffic locality
d >= 4
◦ Harder to build, more wires, longer average length
◦ Fewer hops, better bisection bandwidth
◦ Can handle non-local traffic
17
18
27.06.2008
9

Wire density tends to be very high near the bisection and
low near the perimeter.
2D mesh has uniform density throughout.
It requires also longer links. Cycle time for network
increases as the logarithm of the wire length.
Rich set of topological alternatives with deep relationships
Design point depends heavily on cost model
◦ Nodes, pins, area, ...
◦ Wire length or wire delay metrics favour small dimension
◦ Long (pipelined) links increase optimal dimension
19
20
27.06.2008
10