Port Oversubscription - The Cisco Learning Network

Oversubscription 

Dave McFarland CAE NCE

Port Oversubscription 

Overview of Gen-1/2/3 products 

Gen-3 was released with NX-OS 4.X and is not part of the 

current CCIE SAN blueprint 

Oversubscription it relates to bandwidth 

Oversubscription it relates to configuring ISLs (E 

and TE ports) 

Rate-Mode oversubscription examples 

Presentation_ID © 2010 Cisco and/or its affiliates. All rights reserved. Cisco Public 

2

Port Bandwidth Oversubscription 

On a Line card or fabric switch, this is when the 

available BW of a port is exceeded by the desired 

capacity of the port. 

e.g. On a DS-9016 (16 port line card). 

The slot and port SFP can allow 2 GB line rate on each 

port. 

There is no oversubscription at this level. Oversubscription 

is 1:1 

e.g. On a DS-9032 (32 port line card). 

The slot will only move 20 GB of full duplex traffic, with 

each of the 32 ports trying to run 2 GB. 

There is oversubscription of 3.2:1 (64 GB / 20 GB) 


3

Generation-1 

Gen-1 is 1Gb and 2 Gb 

Gen-1 has specific requirements to dedicate speed. This is commonly used to 

create ISLs. The speed must be set. 

16 port Generation-1 Line Card 

16 port LC, all ports run at full line rate, default Auto 

Ports are Auto select (F, E, etc) 


32 port LC, all ports are oversubscribed, default Fx 

Only 1 port in a port group can be configured as a ISL (E, TE) port 

For an ISL the switchport type must be set E/TE 

20/40 port Generation Fabric Switch 

All port are oversubscribed, default Fx 

Only the 1 st port in a port group can be configured as a ISL (E, TE) port 


4

Generation 2 

Gen-2 is 1/2/4 GB 

Generation 2 is much more flexible. But, with flexibility there is more configuration 

required . 


All ports run at full line rate, default Auto 


18-4, 24, 48 port Generation-2 Line Card 

All ports are oversubscribed, default Fx 

Any port in a port group can be configured as a ISL (E, TE) port. Specific port 

speeds and dedicated rate modes must be followed 

9222i Generation-2 Fabric Switch 

All ports are oversubscribed, default Fx 

Any port in a port group can be configured as a ISL (E, TE) port. Specific port 

speeds and dedicated rate modes must be followed 

9124/9134 Generation-2 Fabric Switch 

All ports run at 4 GB full line rate, default Auto 



5

Generation 3 

Gen-3 is 1/2/4/8 Gb 

Generation 3 is the same as Generation-2 in terms of flexibility. There is 

greater bandwidth in line card/fabric switch compared to generation-2 


44/4 

All port are oversubscribed. 

Any port in a port group can be configured as a ISL (E, TE) port. Specific 

port speeds and dedicated rate modes must be followed 

All port are oversubscribed. 

Any port in a port group can be configured as a ISL (E, TE) port. Specific 

port speeds and dedicated rate modes must be followed 

9248 Generation-3 Fabric Switch 

All ports are 8 GB line rate 

Any port in a port group can be configured as a ISL (E, TE) port. 


6

Generation 2 and 3 Configuration Terminology 

Port Groups – Portion of a switching element (line card or fabric switch) that 

has a subset of the allocated bandwidth 

Ports in a Port Group – Specific ports associated with a port group 

Rate-Mode Shared – Allocated bandwidth shared among ports that are 

configured as shared 

Rate-Mode Dedicated – Allocated bandwidth dedicated to ports that are 

configured as dedicated 

Oversubscription Ratio – Within a Port Group, the ratio of the available 

bandwidth to configured shared ports. There are default values. This can be 

over-ridden on a module basis. 

Out-Of-Service – Port is removed from sharing any resources. This is not the 

same as “shut”. When a port is shut, resources are still allocated to it. 


7

Part Number Description 

Gen-2 Modules 

Port/ 

group groups 

MaxBW/ 

Bandwidth/ port 

group (Gbps) (Gbps) 

Max 

BW/slot 

Dedicated 

Rate mode 

Supported 

DS-X9148 48 port 4 Gbps 12 4 12.8 4 51.2Yes Yes 

Groups per this line card (4) 

12 ports per group 

Switching 

Module Type 

9148 Gen-2 Example 

Configured 

Speed Ratio Enabled Ratio Disabled 

Maximum BW 

Gbps 

48 Port Auto 4 0.8 0.09 4.2 

Shared rate 

mode 

supported 


8

Configuring a Dedicated speed port on a 9148 

(e.g. for an ISL) 

I need to set the rate-mode dedicated and speed to 4000. 

ISLs cannot be shared rate mode. 

The number of ports and the speeds I can set dedicated 

are based on the available bandwidth in a Port Group 

If I run out of bandwidth in a port group, options are: 

Reduce the shared bandwidth on the remaining ports 

Out-of-Service Ports 

Disable oversubscription bandwidth ratio 

Use “show port-resources module X” over remembering 

tables 


9


Switching 

Module Type 

Configured 

Speed 


group groups 

Ratio 

Enabled 

Ratio 

Disabled 

Maximum 

BW Gbps BW/Group 

48 Port Auto 4 0.8 0.09 4 12.8 

Auto 2 0.4 0.045 2 12.8 

1Gbps 0.2 0.0225 1 12.8 

With the ratio-enabled (default setting), you cannot create rate-mode 

dedicated ports that will cause the shared-mode port speed to exceed 

the ratio 

1 port at 4 GB dedicated in a port group 

With 1 x 4 GB dedicated, 12.8 – 1(4) = 8.8 GB remaining. I have 11 

remaining ports. 

At 4 GB shared ports, my ratio is 8.8 GB / 11= 0.88 

Per the table above, this will work. 

MaxBW/ 



Max 

BW/slot 

Dedicated 

Rate mode 

Supported 


Shared rate 

mode 

supported 


10

cae-sj-9506-1(config)# interface fc 2/1 

cae-sj-9506-1(config-if)# switchport rate-mode dedicated 

cae-sj-9506-1(config-if)# switchport speed 4000 

cae-sj-9506-1(config-if)# do show port-resources module 2 

Module 2 

Available dedicated buffers are 4653 

Port-Group 1 

Total bandwidth is 12.8 Gbps 

Total shared bandwidth is 8.8 Gbps 

Allocated dedicated bandwidth is 4.0 Gbps 

-------------------------------------------------------------------- 

Interfaces in the Port-Group B2B Credit Bandwidth Rate Mode 

Buffers (Gbps) 

-------------------------------------------------------------------fc2/1 

16 4.0 dedicated 

fc2/2 16 4.0 shared 








fc2/10 16 4.0 shared 

fc2/11 16 4.0 shared 

fc2/12 16 4.0 shared 


11

Maximum 4 GB dedicated ports in a group 

Switching Configured 

Module Type Speed 

Ratio 

Enabled 

Ratio 

Disabled 

Maximum 



Auto 2 0.4 0.045 2 12.8 

1Gbps 0.2 0.0225 1 12.8 

How many rate mode-dedicated 4 GB ports can I create with the 

remaining ports 4 GB shared 

If I attempt to make one more, it fails 

cae-sj-9506-1(config-if)# interf fc 2/2 


fc2/2: (error) Bandwidth not available 

12.8 – (2 * 4) = 4.8 available 

4.8 / 10 (remaining ports) = .48 ratio 

I must set the remaining shared ports Auto 2 Gb or less for it to work 


12

cae-sj-9506-1(config)# interface fc 2/3-12 

cae-sj-9506-1(config-if)# switchport speed 


cae-sj-9506-1(config-if)# exit 




Module 2 






-------------------------------------------------------------------- 



-------------------------------------------------------------------fc2/1 


fc2/2 16 4.0 dedicated 








fc2/10 16 2.0 shared 

fc2/11 16 2.0 shared 

fc2/12 16 2.0 shared 


13

3 x 4 GB in a Port Group? 



Ratio 

Enabled 

Ratio 

Disabled 

Maximum 



Auto 2 0.4 0.045 2 12.8 

1Gbps 0.2 0.0225 1 12.8 

Can I make 3 ports rate-mode dedicated at 4 GB? 

12.8 – (3 * 4) = 0.8 GB remaining 

0.8 / (9 remaining ports) = 0.08 So, not even if I set the remaining ports 

to 1 GB 

The options at this point are to: 

Out-of-service ports 

Disable the oversubscription limit 


14






fc2/3: (error) Bandwidth not available 

cae-sj-9506-1(config-if)# do sho port-resources module 2 

Module 2 






-------------------------------------------------------------------- 



-------------------------------------------------------------------fc2/1 










fc2/10 16 1.0 shared 

fc2/11 16 1.0 shared 

fc2/12 16 1.0 shared 


15



Out-of-Service 

Ratio 

Enabled 

Ratio 

Disabled 

Maximum 



Auto 2 0.4 0.045 2 12.8 

1Gbps 0.2 0.0225 1 12.8 

I need to place enough ports “Out-Of-Service” to meet the ratio 

requirements 

12.8 – (3 * 4) = 0.8 GB remaining 

0.8 / (9 remaining ports) = 0.08 So, not even if I set the remaining ports 

to 1 GB 

At 1 GB, the ratio must be >= 0.2 

0.8 GB remaining / 0.2 (GB/port ratio) = 4 ports. I must out-of-service the 

remaining 5 ports 


16

cae-sj-9506-1(config-if)# interface fc 2/8-12 

cae-sj-9506-1(config-if)# shut 

cae-sj-9506-1(config-if)# out-of-service 

Putting an interface into out-of-service will cause its shared resource 

configuration to revert to default 

Do you wish to continue(y/n)? [n] y 

cae-sj-9506-1(config-if)# interface fc 2/3 



Module 2 






-------------------------------------------------------------------- 



-------------------------------------------------------------------fc2/1 








fc2/8 (out-of-service) 






17

Disable the default oversubscription ratio 



Ratio 

Enabled 

Ratio 

Disabled 

Maximum 



Auto 2 0.4 0.045 2 12.8 

1Gbps 0.2 0.0225 1 12.8 

With 3 ports dedicated 4 GB, I have 0.8 GB remaining bandwidth 

0.8 / 9 (remaining ports) = 0.09 ratio 

Per the table above I can have the 9 remaining ports in 4 GB shared 

mode if I disable the oversubscription ratio 


18


cae-sj-9506-1(config-if)# switchport rate-mode shared 


cae-sj-9506-1(config-if)# no out-of-service 



cae-sj-9506-1(config-if)# exit 

cae-sj-9506-1(config)# no rate-mode oversubscription-limit module 2 



cae-sj-9506-1(config-if)# do sho port-resources module 2 

Module 2 






-------------------------------------------------------------------- 



-------------------------------------------------------------------fc2/1 










fc2/10 16 4.0 shared 

fc2/11 16 4.0 shared 

fc2/12 16 4.0 shared 


19

Gen-3 Module Details 


Modules > SAN- 

OS 3.3 


group groups 

Bandwidth 

/ 

group 

(Gbps) 

MaxBW/ 

port 

(Gbps) 

Max 

BW/ 

slot 

Dedicated 

Rate mode 

Supported 

DS-X9248-96K9 48 port 8 Gbps 6 8 12.8 8 102.8Yes Yes 

DS-X9224-96K9 24 port 8 Gbps 

4/44 port 8 

3 8 12.8 8 102.8Yes Yes 

DS-X9248-48K9 Gbps 12 4 12.8 8/4 51.2Yes Yes 

Fabric Switch 

SAN-OS > 3.3 

DS-C9148-K9 48 port 8 Gbps 4 6 32 

Line 

8rate 

Yes No 

Shared 

rate mode 

supported 


20



Gen-2 Modules 


group groups 

MaxBW/ 



Max 

BW/slot 

Dedicated 

Rate mode 

Supported 


DS-X9124 24 port 4 Gbps 

18 port 4 Gbps 4 

6 4 12.8 4 51.2Yes Yes 

DS-X9304-18K9 GigE 6 4 12.8 4 51.2Yes Yes 

DS-X9112 12 port 4 Gbps 3 4 12.8 4Line rate Yes No 

DS-X9704 4 port 10 Gbps 1 4 10 10Line rate Yes No 

Gen-2 Fabric Switch 

DS-C9134-K9 


port 10 Gbps 4 8 16 4Line rate Yes No 

1 2 10 10Line rate Yes No 

DS-C9124K9 24 port 4 Gbps 


4 6 16 4Line rate Yes No 

DS-C9222i-K9 GigE spare slot 6 3 13 4 48Yes Yes 

Shared rate 

mode 

supported 


21


Part 

Number Description 

Gen-1 

Modules 

Gen-1 

Fabric 

Switch 

DS-X9016 

DS-X9032 

DS-C9120- 

K9 

DS-C9140- 

K9 

Bandwidth 

/ 

Port/ group 

group groups (Gbps) 

MaxBW 

/ 

port 

(Gbps) 

MaxBW 

/ 

slot 

Dedicate 

d Rate 

mode 

Supporte 

d 

16 port 2 

Line 

Gbps 

32 port 2 

4 4 10 2rate 

No No 

Gbps 4 8 5 2 40No No 

20 port 2 

Gbps 

40 port 2 

4 5 5 2 20No No 

Gbps 4 10 5 2 40No No 

Shared 

rate mode 

supported 


22

Gen-2 Oversubscription Ratios 

Switching 

Module 

Type 

Configured 

Speed 

Ratio 

Enabled 

Maxim 

Ratio um Switching 

DisablBW 

Module 

ed Gbps Type 

Configu 

red 

Speed 

Ratio 

Enable 

d 

Ratio 

Disabled 

Maximum 

BW Gbps 

48 Port Auto 4 0.8 0.09 448 Port Auto 4 0.8 0.09 4 

Auto 2 0.4 0.045 

0.022 

2 Auto 2 0.4 0.045 2 

1Gbps 0.2 5 1 1Gbps 0.2 0.0225 1 

24 port Auto 4 1 0.27 424 port Auto 4 1 0.27 4 

Auto 2 0.5 0.135 2 Auto 2 0.5 0.135 2 

1 Gbps 0.25 0.067 1 1 Gbps 0.25 0.067 1 


23

Gen-2 Configuration Scenarios 

Module 

4 GB 

Dedicated 

2 GB 

Dedicated 

Ports must pass 

some traffic 

Ports out-ofservice 

Oversubscrip 

tion Ratio 

9513- 

Module 6 17-18,22-24 None 13-16,19-21 None As required 

9513- 

Remaining ports 

Module 5 1-3,7-8 

9513- 

None None in groups 1 and 2 As required 

Module 1 none 1-12 None None As required 

9513- 

2-5,8-11,14- 

Module 3 1,7,13 

9506- 

17 Remaining None As required 

Module 1 Any 6 ports None Remaining None Default 

9506- 3 

Module 2 ports/group None Remaining None As required 

9506- 2 

2 

Module 3 port/group ports/group None As required As required 


24

Port Oversubscription - The Cisco Learning Network

Create successful ePaper yourself

Delete template?

Save as template?