ACCA PowerPoint presentation Template and style guide
ACCA PowerPoint presentation Template and style guide
ACCA PowerPoint presentation Template and style guide
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
Congestion Management in Lossless<br />
Interconnects: Challenges <strong>and</strong> Benefits<br />
José Duato<br />
Technical University of Valencia (SPAIN)<br />
Conference title 1
Outline<br />
Why is congestion management required?<br />
Benefits<br />
Congestion <strong>and</strong> congestion management strategies<br />
Challenges<br />
Enhancing reactive congestion management<br />
Congestion management & adaptive routing<br />
HOL blocking elimination techniques<br />
Hybrid congestion management strategy<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 2
Current role of interconnection networks<br />
• For three decades the goal of computer architects has<br />
been to keep the processors busy → top performance<br />
• Interconnects were usually cheap, <strong>and</strong> never a bottleneck<br />
• Now, global system performance in large systems is<br />
limited by the interconnection network (e.g. Tianhe-1A)<br />
• Network latency directly impacts application performance,<br />
<strong>and</strong> network saturation leads to latency increasing by<br />
orders of magnitude<br />
Saturation should be avoided at all costs<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 3
Conflicting interests: cost vs. performance<br />
• Saturation was traditionally avoided by overdimensioning<br />
the interconnection network, but this is becoming very<br />
expensive<br />
• No overdimensioning → Danger when working with high<br />
traffic loads (close to the saturation point)<br />
• Network performance (throughput, latency) should be<br />
good under very different traffic patterns & load scenarios<br />
• Traffic load may significantly vary over time, reaching<br />
saturation<br />
At saturation, network performance drops dramatically<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 4
Network throughput at saturation<br />
HS<br />
starts<br />
HS<br />
ends<br />
HS = traffic injected to Hot Spot destination<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 5
Should we currently care about congestion?<br />
Growing processor<br />
speed<br />
Processor prices<br />
drop (dem<strong>and</strong>)<br />
Congestion<br />
probability<br />
grows<br />
Growing link<br />
speed<br />
Relative<br />
interconnect<br />
cost increases<br />
Saturation point<br />
reached with lower traffic load<br />
Congestion<br />
Performance<br />
Management<br />
degradation<br />
Strategies<br />
Power<br />
consumption<br />
increases<br />
Power management<br />
Smaller networks<br />
B<strong>and</strong>width decreases<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 6
Benefits<br />
• Stable performance when the network reaches saturation<br />
• No performance drop<br />
• Delivers maximum achievable throughput<br />
• Reacts quickly when power management turned some<br />
components off <strong>and</strong> dem<strong>and</strong> suddenly increases<br />
• Prevents performance degradation due to power management<br />
• Enables more aggressive power saving strategies without risk<br />
• Helps to keep performance when faults occur <strong>and</strong> fault<br />
tolerance techniques enable alternative paths<br />
• Alternative paths may become congested (less resources are available)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 7
Contention<br />
• Several packets from different flows request the same<br />
output port in a switch<br />
• One packet makes progress, the others wait<br />
Network<br />
contention<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 8
Congestion<br />
• Persistent contention<br />
• Buffers containing packets belonging to flows involved<br />
in contention become full<br />
Persistent<br />
network<br />
contention<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 9
Congestion propagation<br />
• In lossless networks, congestion is quickly propagated<br />
by flow control, forming congestion trees<br />
Flow control<br />
Persistent<br />
network<br />
contention<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 10
Congestion propagation<br />
• In lossless networks, congestion is quickly propagated<br />
by flow control, forming congestion trees<br />
• Congestion trees may cause Head-of-Line blocking<br />
Congestion<br />
propagation<br />
may reach<br />
the sources<br />
Congestion affects<br />
packets belonging<br />
to flows that do not<br />
cause congestion<br />
Persistent<br />
network<br />
contention<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 11
Congestion trees<br />
• Congestion tree structure:<br />
Congestion<br />
tree leaf<br />
Congestion<br />
tree leaf<br />
Congestion<br />
tree branch<br />
Congestion<br />
tree branch<br />
Congestion<br />
tree root<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 12
Traditional solution<br />
• Overdimensioning the network<br />
• Many more components than really necessary<br />
• Offered network b<strong>and</strong>width is much higher than the<br />
b<strong>and</strong>width requested by end nodes<br />
Overdimensioned network<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 13
Traditional solution<br />
• Overdimensioning the network<br />
• Advantage: low link utilization → low latency<br />
Latency<br />
Latency<br />
Working<br />
zone<br />
Injected traffic<br />
Congestion<br />
zone<br />
Traffic<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 14
Classical congestion management strategies<br />
• Proactive congestion management (congestion<br />
prevention)<br />
• Path setup before data transmission<br />
• Used in ATM, computer networks (QoS)<br />
• High overhead, high setup latencies, poor link utilization<br />
(not suitable for HPC)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 15
Classical congestion management techniques<br />
• Reactive congestion management (congestion<br />
recovery)<br />
• Injection limitation techniques using closed-loop feedback<br />
• Does not scale well with network size <strong>and</strong> link b<strong>and</strong>width<br />
– Notification delay (proportional to distance / number of hops)<br />
– Link <strong>and</strong> buffer capacity (proportional to clock frequency)<br />
– May produce traffic oscillations (closed loop system with pure delay)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 16
Other approaches<br />
• Adaptive Routing<br />
• May help to delay the occurrence of congestion<br />
• Useless when heavy congestion arises<br />
• Problems regarding in-order packet delivery<br />
• Packet dropping<br />
• Not suitable for most current HPC parallel applications<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 17
Challenges<br />
• To develop congestion management techniques that react<br />
locally <strong>and</strong> immediately when congestion arises<br />
• To make congestion management techniques truly scalable<br />
• To achieve coordination among end nodes without explicit<br />
communication among them<br />
• To eliminate instabilities <strong>and</strong> oscillatory responses<br />
• To minimize the number of extra resources needed to h<strong>and</strong>le<br />
congestion<br />
• To make congestion management compatible with adaptive<br />
routing<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 18
Enhancing reactive congestion management<br />
• Stop injecting packets for a while when a BECN is received<br />
• Do not change injection rate again until feedback from<br />
previous changes is received to prevent oscillations<br />
• Source nodes can dynamically adjust their injection rate to<br />
available b<strong>and</strong>width without communicating among them<br />
– Inject exactly one packet when a BECN is received<br />
– New contenders are automatically detected <strong>and</strong> injection rate reduced<br />
– Slightly reduce the above rate to slowly eliminate the congestion tree<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 19
Congestion management & adaptive routing<br />
• Existing congestion management techniques do not<br />
work correctly with adaptive routing<br />
• Injection rate is adjusted for a certain congestion, but now<br />
packets may follow a different path (unstable behavior)<br />
• Adaptive routing may spread congestion over more links<br />
• Never use adaptive routing for congested packets<br />
when the congestion point is at an end node<br />
• In this case, adaptive routing does not help, spreads<br />
congestion over more links, <strong>and</strong> increases HOL blocking<br />
• Use adaptive routing otherwise(more research needed here)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 20
HOL blocking elimination techniques<br />
• Key idea:<br />
The real problem is not the congestion itself, but its negative<br />
effect (HOL blocking)<br />
By eliminating HOL blocking, congestion becomes<br />
harmless<br />
In general, different buffers required at each port for<br />
separating packet flows<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 21
HOL blocking example<br />
33 %<br />
33 %<br />
33 %<br />
33 %<br />
Sw. 1<br />
Sw. 2<br />
Sw. 3<br />
Sw. 4<br />
33 %<br />
33 %<br />
33 %<br />
33 %<br />
Sw. 5<br />
Sw. 6<br />
Sw. 7<br />
33 %<br />
33 %<br />
66 %<br />
Sw. 8<br />
Congested flows<br />
Non-congested flows<br />
100 %<br />
33 %<br />
Dst. 1<br />
Dst. 2<br />
33 % Sending<br />
33 % Stopped<br />
33 % Sending<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 22
Real-life HOL blocking example<br />
The A-31 highway metaphor<br />
A-43<br />
The flow is affected by<br />
the bottleneck of the<br />
A-31 highway<br />
A-31<br />
Bottleneck<br />
Map Source: Google Maps<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 23
HOL blocking elimination techniques<br />
• VOQnet (Virtual Output Queuing at network level)<br />
• A separate queue at each input port for every destination<br />
• Packets with the same destination are stored in the same<br />
queue<br />
• Completely eliminates HOL blocking<br />
• Number of required buffer resources increases at least<br />
quadratically with network size !!!<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 24
HOL blocking elimination techniques<br />
• VOQsw (Virtual Output Queuing at switch level) &<br />
DAMQs (Dynamically Allocated Multi-Queues)<br />
• A separate queue at every input port for every output port<br />
• Packets requesting the same output are stored in the same<br />
queue<br />
• Better than nothing but does not eliminate HOL blocking<br />
completely. Effectiveness depends on traffic pattern.<br />
• Virtual Channels<br />
• Performance depends on channel (queue) assignment<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 25
HOL blocking elimination techniques<br />
• DBBM (Destination-Based Buffer Management)<br />
• Several groups of destinations are defined<br />
• A separate queue for each group at every port<br />
• Packets with destinations in the same group are stored at the<br />
same queue<br />
• OBQA (Output-Based Queue Assignment)<br />
• Suitable for fat-trees with DESTRO routing<br />
• Queue assignment linked with topology & routing algorithm<br />
• Reduces HOL blocking with the minimum number of queues<br />
per port<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 26
OBQA description<br />
Logical input port organization<br />
• Each input port has a number of queues (q) smaller than<br />
switch radix<br />
• OBQA assigns packets to queues using this formula:<br />
Selected_Queue = Requested_Output_Port MOD q<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 27
OBQA evaluation<br />
Uniform traffic simulation results<br />
Network Latency (cycles) vs Normalized<br />
Generated Traffic<br />
4-ary 4-tree<br />
8x8 switches<br />
(configuration #2)<br />
16-ary 2-tree<br />
32x32 switches<br />
(configuration #3)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 28
HOL blocking elimination techniques<br />
• RECN (Regional Explicit Congestion Notification) &<br />
FBICM (Flow-Based Implicit Congestion Management)<br />
• Key differences with respect to previous techniques:<br />
–Explicitly identifies congested points<br />
–Congestion information storage<br />
–Dynamic queue allocation for congested flows<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 29
Principles of RECN-like solutions<br />
• Congestion becomes harmless if the HOL blocking produced by<br />
congested packets is completely eliminated.<br />
• HOL blocking produced by congested packets is completely<br />
eliminated if they are buffered separately.<br />
• Non-congested packets can share queues without suffering<br />
significant HOL blocking.<br />
• Congested packets can be separately buffered by using a small<br />
number of queues per port.<br />
• Congested packets must be explicitly identified (i.e. packets<br />
belonging to flows contributing to create some congestion).<br />
• Precise identification of congested packets is based on previous<br />
knowledge of the location of existing congestion points.<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 30
RECN basic procedure<br />
• Congested points are detected at any input or output switch<br />
port of the network<br />
• The routes to detected congested points are progressively<br />
notified to input <strong>and</strong> output ports crossed by congested flows<br />
• After receiving a notification, a port dynamically allocates a<br />
CAM line to store the location of the congested point, <strong>and</strong> a<br />
set-aside queue (SAQ) to store congested packets<br />
• A packet arriving at a port will be stored in a SAQ if it will<br />
pass through the congested point associated to that SAQ<br />
• A packet arriving at a port will be stored in the st<strong>and</strong>ard<br />
(“cold”) queue if its route does not match any CAM entry<br />
• SAQs can be dynamically deallocated, <strong>and</strong> later allocated for<br />
other congested points<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 31
How RECN works<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 32<br />
A congestion<br />
point forms
How RECN works<br />
Cold queue fills over<br />
a threshold<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 33
How RECN works<br />
Internal notification to<br />
each input sending<br />
packets to the<br />
congested output<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 34
How RECN works<br />
New SAQs are allocated for<br />
packets addressed to<br />
the congested output port<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 35
How RECN works<br />
Notifications sent when<br />
the SAQs fill over a threshold<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 36
How RECN works<br />
A new SAQ is allocated<br />
for the congested output<br />
at each notified output<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 37
How RECN works<br />
Internal notifications when the<br />
SAQs receive packets <strong>and</strong> the<br />
occupancy is over a threshold<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 38
How RECN works<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 39
How RECN works<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 40
How RECN works<br />
At the end, congestion<br />
tree packets are completely<br />
stored in SAQs<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 41
How RECN works<br />
“Cold flow” sharing some<br />
network resources with a<br />
branch of the congestion tree<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 42
How RECN works<br />
Cold packets are never stored in<br />
SAQs, so they never share a<br />
queue with congested packets<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 43
RECN basics<br />
• RECN achieves efficiency <strong>and</strong> scalability in source routing<br />
environments<br />
0 3<br />
… -1 +4 +3<br />
Turnpointer<br />
Turnpool<br />
Packet header information: The routing information is included<br />
in packet header <strong>and</strong> congestion notifications (turnpool), <strong>and</strong><br />
it is used at each hop (turnpointer)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 44
CAM structure<br />
CAM<br />
Valid<br />
v turnpool bit mask b Xoff<br />
v turnpool bit mask b Xoff<br />
v turnpool bit mask b Xoff<br />
Congested point<br />
Xon/Xoff<br />
Flow control<br />
Blocked<br />
SAQ 0<br />
SAQ 1<br />
SAQ n-1<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 45
Reception of packets after SAQ allocation<br />
Turn pointer<br />
4<br />
4 2<br />
Header of incoming packet<br />
+4<br />
The incoming packet<br />
is stored in SAQ0<br />
Cold Queue<br />
SAQ 0<br />
SAQ 1<br />
SAQ n-1<br />
1 …..00004<br />
….0000111<br />
….00001 ? 0<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 46<br />
CAM line SAQ 0<br />
CAM line SAQ 1<br />
CAM line SAQ n-1
FBICM operation<br />
Key features<br />
• Effective HOL blocking elimination in networks with<br />
distributed routing<br />
• Implicit congestion points identification, detecting flows<br />
heading to them just by inspecting packet destination<br />
• Congestion information is based on destinations instead of<br />
turnpools <strong>and</strong> it represents any network congested point<br />
New CAM structure, new detection, propagation<br />
<strong>and</strong> resource management policies<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 47
FBICM operation<br />
IQ Switch architecture<br />
• Normal Flow Queues (NFQ)<br />
• Congested Flow Queues (CFQ)<br />
Separate non-congested flows<br />
from congested flows<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 48
FBICM operation<br />
CAM structure<br />
• Congested flow identification fields<br />
Congested port, Hops to reach, Destination list, Next CFQ (Congested Flow<br />
Queue)<br />
• Flow Control fields<br />
Stop & Go, Sent Stop, Receiving control<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 49
FBICM operation<br />
Congestion detection<br />
Switch 1 CAM allocation Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
Primary CFQ +<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Threshold NFQ<br />
exceeded<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 50<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Congested Packets are<br />
stored in the CFQ<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 51<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Congested Packets are<br />
stored in the CFQ<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 52<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Congested Packets are<br />
stored in the CFQ<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 53<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Congested Packets are<br />
stored in the CFQ<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 54<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 55<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
HOL blocking is<br />
avoided<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 56<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
HOL blocking is<br />
avoided<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Packet Processing<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 57<br />
NFQ<br />
CFQ<br />
CAM<br />
Destination_list<br />
HOL blocking is<br />
avoided<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Congestion Information Propagation<br />
Switch 1 Switch 2<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 58<br />
NFQ<br />
CFQ<br />
Destination_list<br />
CAM 0<br />
NextCFQ: null<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Congestion Information Propagation<br />
New CAM allocation<br />
(copy of CAM 0)<br />
Switch 1 Switch 2<br />
CAM 1<br />
P4<br />
P5<br />
P6<br />
P7<br />
external Stop<br />
New CAM Line Information:<br />
CAM 0<br />
CAM 1<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Destination_list<br />
The threshold is<br />
exceeded in the CFQ<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 59<br />
NFQ<br />
CFQ<br />
CAM 0<br />
NextCFQ: null<br />
Active Cong_Port: P6 Hops: 1 Destination_list NextCFQ: 0 Stop: true …<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Congestion Information Propagation<br />
Switch 1 port <strong>and</strong> matches with congestion Switch 2<br />
Internal Stop<br />
CAM 1<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
CAM 0<br />
Active<br />
Cong_Port: P6<br />
Congested packet reaches output<br />
information in CAM 1<br />
Hops: 1<br />
Destination_list<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 60<br />
NFQ<br />
CFQ<br />
CAM 0<br />
NextCFQ: null<br />
CAM 1 Active Cong_Port: P6 Hops: 1 Destination_list NextCFQ: 0 Stop: true …<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Congestion Information Propagation<br />
Switch 1 updating congested point Switch 2<br />
NFQ<br />
CFQ<br />
CAM 2<br />
Internal Stop<br />
CAM 1<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
CAM 0<br />
Active<br />
Cong_Port: P6<br />
New CAM + CFQ allocated<br />
information<br />
Hops: 1<br />
Destination_list<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 61<br />
NFQ<br />
CFQ<br />
CAM 0<br />
NextCFQ: null<br />
CAM 1 Active Cong_Port: P6 Hops: 1 Destination_list NextCFQ: 0 Stop: true …<br />
CAM 2 Active Cong_Port: P6 Hops: 2 Destination_list NextCFQ: 1 Stop: true …<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM operation<br />
Congestion Information Propagation<br />
Switch 1<br />
NFQ<br />
CFQ<br />
CAM 2<br />
Congestion tree resources<br />
will be released<br />
dynamically<br />
CAM 1<br />
P4<br />
P5<br />
P6<br />
P7<br />
New CAM Line Information:<br />
CAM 0<br />
Active<br />
Cong_Port: P6<br />
Hops: 1<br />
Congestion tree<br />
branch<br />
Destination_list<br />
Switch 2<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 62<br />
NFQ<br />
CFQ<br />
CAM 0<br />
NextCFQ: null<br />
CAM 1 Active Cong_Port: P6 Hops: 1 Destination_list NextCFQ: 0 Stop: true …<br />
CAM 2 Active Cong_Port: P6 Hops: 2 Destination_list NextCFQ: 1 Stop: true …<br />
…<br />
P4<br />
P5<br />
P6<br />
P7
FBICM evaluation<br />
Network Throughput vs. Load (Config. 1)<br />
R<strong>and</strong>om Uniform Traffic<br />
BMIN 64 x 64<br />
Congested Traffic<br />
BMIN 64 x 64<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 63
FBICM evaluation<br />
Network Throughput vs. Time (Config. 1)<br />
Real Traffic (CF = 20)<br />
BMIN 64 x 64<br />
Real Traffic (CF = 40)<br />
BMIN 64 x 64<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 64
Hybrid congestion management strategy<br />
Key ideas<br />
• Use FBICM to quickly <strong>and</strong> locally eliminate HOL blocking,<br />
propagating congestion information <strong>and</strong> allocating buffers as<br />
necessary<br />
• Use reactive congestion management to slowly eliminate<br />
congestion, deallocating FBICM buffers whenever possible<br />
Use of FBICM provides immediate response <strong>and</strong><br />
allows reactive congestion management to be tuned<br />
for slow reaction, thus avoiding oscillations<br />
Reactive congestion management drastically reduces<br />
FBICM buffer requirements (just one buffer per port)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 65
Conclusions<br />
• Interconnects performance : Key when power management enabled<br />
• Goal: To achieve best possible behavior with limited number of resources<br />
• Congestion (HOL blocking): Serious menace<br />
• DBBM <strong>and</strong> OBQA reduce HOL blocking in specific scenarios (ad-hoc<br />
techniques ) with a small set of resources<br />
• Reactive congestion management: Does not scale well. Can be improved<br />
• RECN: efficiently eliminates HOL blocking in Source Routing Networks<br />
• FBICM: efficiently eliminates HOL blocking in Distributed Deterministic<br />
Routing Networks<br />
• Hybrid congestion management: Mechanisms help each other<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 66
Acknowledgements<br />
• Pedro García (University of Castilla – La Mancha) developed the research<br />
on RECN <strong>and</strong> FBICM under my guidance <strong>and</strong> prepared part of the slides in<br />
this <strong>presentation</strong><br />
• The techniques to enhance reactive congestion management are being<br />
developed in collaboration with Simula Research Laboratory (Oslo)<br />
HPC Advisory Council Workshop. March 21-23, 2011 - Congestion Management 67
Thanks!!<br />
Any question?<br />
Conference title 68