PCIe Devices - PLX Technology

Presents 

PCI Express Overview 

By Pamela Frinzi 

Copyright by Dashcourses Inc, 2009

Introduction 

n 

n 

n 

Objectives 

Specifications Covered 

Presentation Layout 

Copyright by Dashcourses, Inc. 2009 

Intro-2

What This Presentation Is About 

n 

The basic PCIe architectural components and their 

interactions 

l 

PCIe high speed serial interconnect, usage, and operations 

– PCIe terminology 

• Fabric, ports, links, paths, packets 

– The PCIe protocol stack 

• Transaction Layer, Data Link Layer, and Physical Layer requirements, 

operations, and header formats 

– Understand the concept of ‘virtual channels’ and the PCIe term 

‘differentiated services’ 

• Bridging, posting, and arbitration 

• Effects on latency and ‘differentiated services’ 

– PCIe configuration requirements 

• PCIe required and optional register sets 


Intro-3

What This Presentation Is About (continued) 

n 

PCIe is an extension of the basic PCI (Peripheral 

Component Interface) specification 

l 

l 

A brief review of the PCI architecture will be provided 

– System terminology, operation, and usage 

– Basic PCI and PCI-X protocol, bus operations, bus arbitration, and 

commands 

– Bridging in PCI 

– PCI device configuration space and configuration space access 

PCI and PCI-X will be discussed as it relates to PCIe and 

compatibility requirements 


Intro-4

Specifications 

n 

The current PCIe specifications as published by the 

PCI Special Interest group or PCI Sig are 

l PCIe 2.0 (December 20, 2006) 

– Supporting specifications 

• PCI Express Card Electromechanical Specification, Revision 1.12.0 

• PCI Express to PCI/PCI-X Bridge Specification, Revision 1.0 

• PCI Express Mini Card Electromechanical Specification, Revision 1.1 

• PCI Local Bus Specification, Revision 3.0 

• PCI-X Addendum to the PCI Local Bus Specification, Revision 2.0 

• PCI Hot-Plug Specification, Revision 1.1 

• PCI Standard Hot-Plug Controller and Subsystem Specification, 

Revision 1.0 

• PCI-to-PCI Bridge Architecture Specification, Revision 1.2 

• PCI Bus Power Management Interface Specification, Revision 1.2 

• Advanced Configuration and Power Interface Specification, Revision 

2.03.0b 

• Guidelines for 64-bit Global Identifier (EUI-64) Registration Authority 


Intro-5

Standards Body 

n 

Pioneered by Intel, the PCI specification has 

become a ‘defacto’ industry standard 

l 

Controlled by the PCI Special Interest Group or PCI SIG 

PCI special interest group 

2575 N.E. Kathryn #17 

Hillsboro, Oregon 97124 

800-433-5177 (USA) 

503-693-6232 (international) 

503-693-8344 (fax) 

pcisig@pcIsig.com 

Http://www.pcisig.com 


Intro-6

Presentation Layout 

Section 1 - PCI Evolution and Architectural Overview 

Section 2 - PCIe Transaction Layer 

Section 3 - PCIe Data Link Layer 

Section 4 - PCIe Physical Layer 


Intro-7

Section 1 

PCI Express Evolution and Architectural Overview 

Copyright by Dashcourses Inc, 2009 

Copyright by Dashcourses, Inc. 2009

PCI Specification History 

n 

n 

n 

PCI Local Bus Specification 

l Version 1.0 released by Intel TM June 22, 1992 

l Version 2.0 released April of 1993 

– Version 2.1 released 1st quarter 1995 

– Version 2.2 released February of 1999 

– Version 2.3 released March 29, 2002 

l Version 3.0 released February 3, 2004 

PCIe is an extension of the PCI Local Bus Specification 

l Version 1.0 released July 22, 2002 

– Version 1.0a released April 15, 2003 

l Version 1.1 released March 28, 2005 

l Version 2.0 released December 20, 2007 

PCI is not a ‘PC’ standard 1 , but is a computing I/O 

specification 

1 PCI is not a standards body specification, it is a vendors consortium specification. 


1-9

PCI I/O Bandwidth Evolution 

10,000 

2.5 and 5.0 GT/s 

Packetized 

PCIe 

Bandwidth (MB/s) 

1,000 

100 

10 

12 MHz 

32-bit 

EISA 

33 MHz 

32/64-bit 

PCI 

66 MHz 

32/64-bit 

PCI 

66/100/133/266/512 MHz 

32/64-bit 

PCI-X 

1 

ISA 

6 MHz 

16-bit 

1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 


1-10

PCIe Compatibility and New Features 

n 

PCIe is backwards compatible at the binary level 

with PCI and PCI-X 

l 

Configuration space, commands, and device access are 

indistinguishable 

– PCIe adds a mandatory new register set and several optional 

register sets 

• Provides foundation for most of PCIe new features 

n 

PCIe hardware interconnect is entirely new 

l 

l 

High speed serial interconnect 

– Can be striped providing speed enhancements (scaling) 

• Signaling enhancements are planned 

Introduces a three layer protocol stack 

– Provides end-to-end, point-to-point, and electrical service 

• Protocol stack mostly built in hardware and silicon 


1-11

PCIe Interconnects – Chip-to-Chip 

CPU 

CPU 

PCIe Serial Links 


DVD 

MONITOR 

AGP 

Chip 

Set 

System 

Memory 

Camera 


1-12

PCIe Interconnect – Chip-to-Board 

CPU 

CPU 



DVD 

MONITOR 

AGP 

Chip 

Set 

System 

Memory 

Camera 

PCIe Serial 

Links 

PCI Bridge 1 

PCI Embedded 


NIC 

Devices 1 Connectors 

PCI Bus 

PCI Connectors 1 

1 

Referred to as ‘Legacy Devices’ 


1-13

PCIe Interconnect – External Connections 


(specification released January 2007) 

PCIe Device 

PCIe Switch 

PCIe Device PCIe Device PCIe Device 


1-14

PCIe Design Possibilities 

Embedded Devices 

CPU 

CPU 

Video 

Graphics 

Root Complex 

Memory 

Switch 

Switch 

Sound System 

Synthesizer 

SCSI 

Controller 

Ethernet 

Controller 

Bridge 

USB 

Controller 

Multiple 192 Kbps Channels 

PCIe Add-In Connectors 

PCI/PCI-X Add-In Connectors 


1-15

PCIe Protocol Stack Logical Layering 

Application and 

System Software 



Transaction Layer 


Data Link Layer 

Physical Layer 

Logical Sub-block 


Specification 




Physical Sub-block 


TX 

RX 

Packet 

TX 

RX 

Packet 


1-16

Compatibility with Existing PCI Specification 

n 

n 

Ability to enumerate and configure PCIe hardware 

using PCI system configuration software (OS) with 

no modifications 

l 

l 

l 

PCI devices (across a PCIe-PCI bridge) must be accessible by 

existing OSs and device drivers 

PCIe add-in card must be accessible to existing OSs and 

capable of being configured 

Compatibility includes 

– Boot over existing OS 

– Support existing I/O device drivers (at binary level) 

– Support existing applications 

New software required to configure/enable new 

PCIe functionality by adopting the PCI 

configuration paradigm 


1-17

Low Latency and High Bandwidth 

n 

n 

n 

Low-overhead 

l 

Maximize application payload bandwidth, and link efficiency 

– Application and/or local OS determines required mix 

Low-latency communications 

l 

Maximum allowable delay or processing time through link 

devices within the fabric 

– Can calculate (deterministic) delay on end-to-end basis within 

the fabric 

High speed serial interconnect 

l 

l 

l 

2.5 and 5.0 Gbps wire speed – called GT/s or giga transfers 

per second 

– Can be aggregated 

Low pin count per device and connector interface 

Longer connection runs 

– Traces on PCBs or cables external 


1-18

New Features 

n 

Improved data integrity 

l 

l 

Link-level data integrity for all types of transactions and 

packets 

– Sequence Number and Link CRC (LCRC) within data link layer 

header provided 

End-to-end CRC (ECRC) data integrity for high availability 

solutions 

n 

Error handling 

l 

l 

Legacy PCI-level error handling 

Advanced error reporting and handling for improved fault 

isolation and recovery 


1-19

New Features (continued) 

n 

Ability to differentiate services (Quality of 

Service or QoS) 

l 

l 

l 

l 

Configurable arbitration policy within every component 

End-to-end QoS tag with each packet 

Support for isochronous traffic 

Packet prioritization 

n 

Hot Plug and Hot Swap Support 

l 

l 

Support for legacy PCI 

Software model for all form factors 


1-20

Scalable Topologies 

n 

Hierarchies using PCI bridge and switch 

configurations 

l 

l 

PCIe-PCI bridge 

– Box-to-box connection to legacy PCI based machines 

– PCIe to PCI or PCI-X bus bridging 

Network type capability with point-to-point serial 

connections by way of PCIe switches 

n 

Ability to aggregate ‘Lanes’ to increase 

bandwidth through core hierarchical fabric 

components 

l 

Greater bandwidth by ‘striping’ PCIe Links 


1-21

PCI and PCI-X Commands 

n 

100% binary compatibility with existing PCI/PCI-X 

C/BE[0:3]# 

0000b 

0001b 

0010b 

0011b 

0100b 

0101b 

0110b 

0111b 

1000b 

1001b 

1010b 

1011b 

1100b 

1101b 

1110b 

1111b 

PCI 

Command 

Interrupt Acknowledge 

Special Cycle 

I/O Read 

I/O Write 

Reserved 

Reserved 

Memory Read 

Memory Write 

Reserved 

Reserved 

Configuration Read 

Configuration Write 

Memory Read Multiple 

Dual Address Cycle 

Memory Read Line 

Memory Write and 

Invalidate 

PCI-X 

Command 

Interrupt Acknowledge 

Special Cycle 

I/O Read 

I/O Write 

Reserved 

Device ID Message 1 

Memory Read DWord 

Memory Write 

Alias to Memory Read Block 

Alias to Memory Write Block 

Configuration Read 

Configuration Write 

Split Completion 1 

Dual Address Cycle 

Memory Read Block 1 

Memory Write Block 1 

Length 

DWord 

DWord 

DWord 

DWord 

N/A 

Burst 

DWord 

Burst 

Burst 

Burst 

DWord 

DWord 

Burst 

N/A 

Burst 

Burst 

1 Commands added in PCI-X 


1-22

PCI and PCI-X Architecture and Command Execution 

Host System 

PCI Device 

Device Function 

Configuration R/W 


Memory R/W 

I/O R/W 

Requestor 

(Completer) 

Completer 

(Requestor) 

Memory R/W 

I/O R/W 

Messages 

Messages 

Host-to-PCI 

Bus Bridge 

PCI Bus 

Device Interface 

PCI Bus 0 


1-23

PCIe Commands 

TLP Type 

Fmt 

[1:0] 

Type 

[4:0] Description 

MRd 

00 and 01 

0 0000 

Memory Read Request 

MRdLk 

00 and 01 

0 0001 

Memory Read Request Locked 

MWr 

10 and 11 

0 0000 

Memory Write Request 

IORd 

00 

0 0010 

I/O Read Request 

IOWr 

CfgRd0 

CfgWr0 

CfgRd1 

CfgWr1 

Msg 

MsgD 

10 0 0010 I/O Write Request 

00 0 0100 Configuration Read Type 0 

10 0 0100 Configuration Write Type 0 

00 0 0101 Configuration Read Type 1 

10 0 0101 Configuration Write Type 1 

01 1 0r 2 

r 1 

r 0 

Message Request, sub-field r[2:0] 

Specifies the Message routing mechanism 

11 1 0r 2 

r 1 

r 0 

Message Request with data payload, sub-field r[2:0] 

Specifies Message routing mechanism 


1-24

PCIe Command Execution 

Host System 





Memory R/W 

Requestor 

Completer 

Memory R/W 

I/O R/W 

I/O R/W 

Messages 

Messages 

Root Complex 







PCIe Link 

PCIe Packet 

PCIe Transaction 


1-25

PCIe Link is a Point-to-Point Connection 

n 

PCIe Link connection is a dual-simplex channel 

between two components 

l 

l 

Components can be endpoints, bridges, or switches 

An PCIe Link consists of two, low-voltage, differentially 

driven signal pairs 

– One transmit pair (simplex) and one receive pair (simplex) 

• Data encoded in formatted packets 

Serial Packets 

Device 

A 

Device 

B 



Packets 


1-26

PCIe Transactions and Packets 

n 

PCIe transactions consist of one or more serial 

packets 

l 

Packets can vary in size depending on the command 

– Information about the transaction is defined in the fields in the 

PCIe packet headers 

• Packets may or may not have a payload and CRC(s) 

– Two basic packet types (Transaction Layer packets or TLPs 

and Data Link Layer packets or DLLPs ) 

Physical/Data Link 

Header 


Header 

Payload ECRC LCRC 

11010110100100010101101001011010001011100010111010001011010010111101011101111101101001011 

Device 

A 




Device 

B 


1-27

PCIe Packet Delineation or Framing 

n 

PCIe packets are detected or delineated at 

receiver by matching predefined symbols 

l 

l 

l 

Referred to as framing symbols 

Detection of predefined symbol sets determines where the 

first bit of the packet is and how the packet will be decoded 

– Received as a TLP or DLLP 

Beginning and ending packet symbols are required for all 

validated packets 

Physical/Data Link 

Header 


Header 

Payload ECRC LCRC 

11010110100100010101101001011010001011100010111010001011010010111101011101000101111001010000001011010110101 

Device 

A 




Device 

B 

Packet Framing Symbols 


1-28

PCI Devices 

n 

n 

n 

PCI is based on a defined bus structure 

One or more PCI compliant devices attached to 

the PCI bus 

l 

Each PCI device may contain up to eight PCI functions 

– PCI function - a logical device 

• For example a sound card, a video card, or an IDE controller 

– Devices control 1-8 logical functions 

Devices may be a master or a target 

l 

l 

l 

Masters may initiate a bus transaction 

– Require a request/grant (REQ#/GNT#) pair wired to an arbiter 

for each master on the bus 

Targets may not initiate a bus transaction 

– Exception is a PCI-X target completing a split transaction 

response 

Devices attach to PCI busses by way of bridges 


1-29

PCI/PCI-X Device/Function 

PCI Single Function Device 

PCI Bus 

PCI Bus and 

Control Signals 

UART 

(Function) 

I/O 

PCI Multiple Function Device 

PCI Bus 

PCI Bus and 

Control Signals 

FDDI 

(Function) 

SCSI 

(Function) 

IDE Bus 

SCSI Bus 


1-30

PCI Topology with Multiple Host/PCI Bus Bridge 

Processor Processor Processor 

Bus=0 

Subor=2 

Host/PCI 

Bridge 1 

Main 

Memory 

Host/PCI-X 

Bridge 

Bus=3 

Subor=4 

PCI Bus 0 

PCI/PCI 

Bridge 

Pri=0 

Sec=1 

Subor=2 

Pri=3 

Sec=4 

Subor=4 

PCI-X/PCI-X 

Bridge 

PCI Bus 3 

Pri=1 

Sec=2 

Subor=2 

PCI/PCI 

Bridge 

PCI Bus 1 

PCI Bus 4 

PCI Bus 2 

1 Only 1 Host/PCI bus numbered ‘0’. Initial 

starting point for bus enumeration. 


1-31

PCIe Devices 

n 

Same definition as for PCI and PCI-X devices 

with the following additions/modifications 

l 

l 

PCIe devices must be compliant with ‘legacy’ operating 

system plug-and-play software utilities 

– Means PCI enumeration, configuration, device driver, 

application, and, if applicable, interrupt utilities 

– PCIe devices (accept the Root Complex) must implement PCI 

required and optional configuration space 

• May optionally implement PCIe optional configuration space 

Connections between PCIe compliant devices is now over a 

serial link 

– Communications over the link is controlled by an PCIe three 

layer protocol stack 


1-32

PCIe Device/Function 

PCIe Single Function Device 


Link 


Protocol Stack 

1G NIC 

(Function) 

I/O 

PCIe Multiple Function Device 


Link 



IDE 

(Function) 

SCSI 

(Function) 

I/O 

SCSI Bus 


1-33

PCIe Topology Originates from a Root Complex 

PCIe Root Complex 



PCIe Device PCIe Device PCIe Switch 





n 

Root Complex defines the origin of the root hub 

l All interconnects are hierarchical, point-to-point connections 

l 

All devices in the hierarchy constitute the PCIe ‘fabric’ 


1-34

PCIe System Fabric 









n 

The PCIe specification defines the rules, behaviors, 

requirements, and options for all devices within the fabric 


1-35

PCIe Port Fabric 









n 

Each Root Port in the Root Complex defines its own fabric 

hierarchy 


1-36

PCI/PCI-X/PCIe Bridges 

n 

Bridges are the mechanism that PCI devices use 

communicate to the host system resources, to 

other PCI devices, and to other PCI busses in the 

system 

n 

Several bridge types have been defined 

l 

l 

Host/PCI and host/PCI-X bridges 

– If multiple host bridges in system, one must be hardwired to 

indicate access to PCI bus 0 

– PCI/PCI (P-P bridge), PCI-X/PCI-X, and PCI/PCI-X bridges are 

defined 

The PCIe root complex is the equivalent to the PCI host-to- 

PCI bus bridge, only it implements the PCIe added 

functions, the PCIe protocol stack, and the PCIe serial link 

– PCIe switches are defined in terms of P-P bridges 


1-37

PCIe and Bridges 









n 

A root complex and PCIe switches are P-P bridge structures 

l Every port connection from a switch or root complex logically 

operates as a P-P bridge 


1-38

PCI/PCI-X Arbitration Example 

n 

n 

Four PCI devices are plugged into a PCI bus 

l 

l 

Devices A and B are grouped as high priority masters 

Devices X and Y are low priority masters 

If all masters make requests at same time, the 

sequence in which grants are made may be as 

follows 

Device Execution Priority 

Arbiter 

l A - first 

l B - second 

l X - third 

l A - fourth 

l B - fifth 

l Y - sixth 

A 

B 

X 

PCI Bus 

Y 


1-39

PCIe Device and Bridge Processing 

n 

PCI bridge processing is based on ordering rules 

l 

l 

Based on transaction type 

– Rules on how, when, and if READ can pass WRITE operations 

– Which transactions are buffered or posted in the bridge 

– Whether operations can be combined or collapsed 

All transactions processed equally based on ordering rules 

Transmit Buffer 

TLP Read 

TLP Write 

DLLP 

PM Message 

Queued Packets 

Which packet goes first ??? 


1-40

PCIe Processing and Priorities 

n 

PCIe is based on the same bridge processing 

rules and adds the capability to ‘differentiate’ 

transactions in several ways 

l 

Differentiation provided by a ‘Traffic Class’ or TC label in a 

PCIe packet 

– TC’s are assigned to ‘Virtual Channels’ or VCs 

– Bridge buffer space and control logic must be provided for 

each VC assigned in the system 

n 

Arbitration is based on P-P bridge processing 

and ordering rules on a per VC basis 

l 

Arbitration on one VC is independent of arbitration on any 

other VC 


1-41

PCIe Switch Arbitration and Virtual Channels 

PCIe Switch Port 

Port arbitration within 

a VC ingress port 

VC arbitration for 

a egress port 

RX 

TX 

TC/VC 

Mapping 

VC0 P1 

Arbitration 

VC0 

Arbitration 

RX 

TX 

VC0 Pn 

VC1 

VC0 

RX 

TX 

TC/VC 

Mapping 

VC1 P1 

Arbitration 

VC1 Pn 

VC1 

These structures and 

queues are replicated 

for each egress port 

VC = virtual channel 


1-42

Mythical Example VC ID and Priority Order 

VC 

Resource 

VC ID 

Extended VC count = 7 

8 th VC 

VC7 

HIGH 

7 th VC 

6 th VC 

5 th VC 

4 th VC 

VC6 

VC5 

VC4 

VC3 

Priority Order 

Strict 

Priority 

For Isochronous Traffic 

Lo Priority Extended VC count = 3 

(Defined by software in port VC 

capability register 1) 

3 ed VC 

2 nd VC 

VC2 

VC1 

Governed by 

VC Arbitration 

Capability field 

1 st VC 

VC0 

LOW 


1-43

PCIe Traffic Differentiation 

n 

Separate data flows can be assigned different 

TC/VC combinations 

l 

If priorities are assigned, switch arbitration rules are 

handled on a per VC basis 




VC0/TC0 Packets 









1-44


n 

A Root Complex (RC) denotes the root of an I/O 

hierarchy that connects a CPU/memory 

subsystem to the I/O subsystem 1 

l 

A Root Port is a virtual PCI/PCI bridge that originates a 

PCIe Hierarchy domain from a Root Complex 

1 In PCI this would be the ‘host/PCI’ bus bridge providing access to bus 0 


1-45

PCIe Root Complex Example 

CPU CPU CPU CPU 

Links originating from the 

Root Complex are 

associated with the fabric 

belonging to that machine 


Endpoint 

Chip Set 

Root Complex 

System 

Memory 

PCI-PCIe Bridge 

Switch 

PCI/PCI-X 

Legacy Devices 

Legacy Endpoint 

PCIe Endpoint 


1-46

PCIe Root Complex Model 

Host system’s ‘front side’ bus 

Root Complex 

Register Block 

Host/PCI Bridge 

PCI Bus #0 

PCI/PCI Bridge 

Configuration 

Registers 

Device #X on 

PCI Bus #0 

PCI/PCI Bridge 

Configuration 

Registers 

Device #X+1 on 

PCI Bus #0 

PCI /PCIe Interface 

PCI /PCIe Interface 

PCIe Links 

Device Endpoint 

Device Endpoint 

PCI Bus #N, device #0 PCI Bus #N+1, device #0 


1-47

PCI IDSEL and Host/PCI Bridge Implementation 

Host 

Processor 

IDSEL decoder 

designed into bridge and 

routed on individual traces 

to PCI devices 

Host/PCI Bridge 

PCI 

Device 1 

I 

D 

S 

E 

L 

PCI 

Device 2 

I 

D 

S 

E 

L 

PCI 

Device 3 

I 

D 

S 

E 

L 


1-48

PCIe Switch Structure 

Upstream Port 

Connection made to 

Root Complex or towards Root Complex 


PCI/PCI 

With Bridge 

Configuration Space 


Virtual PCI Bus 

PCI/PCI 

With Bridge 


PCI/PCI 

With Bridge 


PCIe Links 

Downstream Port 


device away from Root Complex 



device away from Root Complex 

Switches in PCIe are multi-port PCI/PCI bridges 

Links are given PCI bus numbers; devices addressed as in PCI by 

bus number, device number, and function number 


1-49

PCIe Arbitration 

n 

In all Root Complex devices and switches, 

arbitration capabilities must be provided 

l 

l 

l 

Can be any method as previously used in PCI 

– Round robin, weighted round robin, or a combination 

Can optionally implement time-based arbitration 

– Used for isochronous applications 

Can assign priorities to any and all of the above on an 

individual platform basis 

n 

Capabilities will be heavily dependent on vendor 

chip set capabilities, as the Root Complex will be 

one of the built-in functions of the chip set 

l 

Usually comes with vendor defined power-up settings, but 

may be re-configured by software 


1-50

PCIe Interrupts 

Host System Bus 

When INTx asserted 

bridge sends message 

to the Root Complex 

identifying the source 

Root 

Complex 

PCI/Express 

Bridge 

System 

Memory 

Express 

Connectors 

PCI Bus 

INTA 

INTB 

INTC 

INTD 

PCI 

Connectors 

INTA 

INTB 

INTC 

INTD 


1-51

PCIe Compatibility with PCI Error Methods 

n 

PCIe requires all devices to have PCI 

configuration space 

l 

Registers will have same definitions and field formats as in 

PCI plus a new PCIe Capabilities register 

– All new required PCIe error capabilities, where appropriate, will 

be mapped to appropriate PCI error Status register bits 

• All required PCIe error Capabilities will also be set in the PCIe 

register set Status and Capabilities fields 

– Optionally an PCIe device may implement the Advanced Error 

Logging and Reporting register set 

n 

Error capabilities are very system dependent and 

discussed in another chapter 


1-52

PCIe Link is a Point-to-Point Connection 

n 

PCIe Link connection is a dual-simplex channel 

between two components 

l 

l 

Components can be endpoints, bridges, or switches 

An PCIe Link consists of two, low-voltage, differentially 

driven signal pairs 

– One transmit pair (simplex) and one receive pair (simplex) 

• Data encoded in formatted packets 


Device 

A 

Device 

B 



Packets 


1-53

PCIe Dual Simplex x1 Link 



Tx 

D+ 

D+ 

Rx 

D- 

D- 

To/from PCIe device 

Physical Layer 8B/10B 

encoding/decoding logic 

To/from PCIe device 

Physical Layer 8B/10B 

encoding/decoding logic 

Rx 

D+ 

D- 

D+ 

D- 

Tx 

n 

Transmitters drive, and receivers must detect a 0.8 to 

1.2 V peak-to-peak signal when driven into a 50 ohm load 

l 

l 


Each dual-simplex pair form a x1, 2.5 or 5 GT/s, connection referred to as 

a Lane 

PCIe Links are composed of one or more Lanes, each Lane 

1-54

PCIe Differential Drivers 

n PCIe defines two analog signals as D+ and D- 

l Peak-to-peak differential determines whether it is a ‘1’ or ‘0’ 

D+ 

Logical 1 

V D+ 

D- 

D+ 

D- 

V D- 

V DIFF 

V DIFF 

Logical 0 


1-55

PCIe Lane Attributes 

n 

Key design attributes for a PCIe Lane are: 

l 

Basic Lane 

– Dual unidirectional differential Links (Transmit and Receive) 

– Data clock embedded using 8B/10B encoding 

• Maximum data throughput is 2 Gbps (enhancements planned) 

l 

Signaling 

– Once initialized, each PCIe Link must operate at one of the 

supported signaling levels 

• Only currently defined signaling level is 2.5 Gbps/Lane/direction 

of raw bandwidth (enhancements planned) 

n 

A PCIe Link is composed of one or more lanes 


1-56

PCIe Link Attributes (continued) 

n 

Key design attributes for a PCIe Link are: 

l 

l 

Lanes 

– A Link must support at least one Lane 

• Each Lane represents a set of differential signal pairs (Transmit 

and Receive) 

– A Link may aggregate multiple Lanes denoted by xN 

• Currently supported values of N are 

– x1, x2, x4, x8, x12, x16, and x32 

– A maximum of 80/160 Gbps of raw bandwidth in each 

direction 

Initialization 

– Link initialized in hardware (no firmware or OS software) 

– Link set up follows a negotiation of Lane widths and frequency 

of operation by agents embedded at each end of the Link 

l 

Symmetry 

– Each Link must support a symmetric number of Lanes in each 

direction 


1-57

Links vs. Lanes 

n 

PCIe Links are made up of one or more Lanes 

l 

l 

Links are configured when devices are attached 

– May optionally be reconfigured by software 

x1, x2, x4, x8, x12, x16, and x32 currently allowed 

configurable Links 

One x4 Link 

Four x1 Links 

TX1-RX1 

TX2-RX2 

TX3-RX3 

TX4-RX4 

TX1-RX1 

TX2-RX2 

TX3-RX3 

TX4-RX4 

Lane 0 Lane 1 Lane 2 Lane 3 

Link A 

Lane 0 

Link A 

Lane 1 

Link B 

Lane 2 

Link C 

Lane 3 

Link D 


1-58

PCIe Can be Aggregated 

n 

PCIe devices may contain multiple x1 lanes that 

may be configured as one or more PCIe Links 

Lane 0 

x1 Link 

Lane 1 

Lane 2 

x1 Link 

x1 Link 

x4 Link 

Device 

A 

Lane 3 

Lane 4 

x1 Link 

x1 Link 

x8 Link 

Lane 5 

x1 Link 

Lane 6 

x1 Link 

Lane 7 

x1 Link 


1-59

Need for I/O Bandwidth 

Bandwidth (MB/s) 

10,000 

1,000 

100 

10 

1 

ISA 

12 MHz 

32-bit 

EISA 

6 MHz 

16-bit 

33 MHz 

32/64-bit 

PCI 

66 MHz 

32./64-bit 

PCI 

66/100/133/266/512 MHz 

32/64-bit 

PCI-X 

32-Bit Transfers 

33 = 1.056 Gbps 


33 = 2.012 Gbps 

x1 = 2.5 or 5 GT/s 

x4 = 10 or 20 GT/s 

x8 = 20 or 40 GT/s 

x16 = 40 or 80 GT/s 

x32 = 80 or 160 GT/s 


2.5 or 5 GT/s 

Packetized 


66 = 2.11 Gbps 

100 = 3.2 Gbps 

133 = 4.256 Gbps 

266 = 8.44 Gbps 

512 = 16.384 Gbps 


66 = 4.21 Gbps 

100 = 6.4 Gbps 

133 = 8.5 Gbps 

266 = 16.384 Gbps 

512 = 32.768 Gbps 

1986 1988 1990 1992 1994 1996 1998 2000 2002 2004 


1-60

MBps vs. Gbps?? 

n 

Comparing data rate exchange capabilities between 

parallel based and serial based technologies is 

difficult 

l 

l 

l 

l 

Parallel – usually expressed in M (10 6 ) or G (10 9 ) bytes per 

second 

Serial – usually expressed in the same terms only in bits per 

second 

Examples 

– Which technology transfers data faster 

• 66 MHz 32-bit PCI bus or PCIe over a 4x link or a 1 Gbps Ethernet 

link?? 

– Is a parallel bus better for multi-media applications 

or for data base type applications?? 

Analysis is difficult 

– Must account for wait states, latency, delay, bus or link utilization, 

arbitration scheme, protocol overhead (present in parallel and 

serial based systems) 


1-61

PCI Configuration Address Space 

n 

n 

n 

Most processors have a linear range of 

addressable read/write memory capability 

l Such as 0-64k, or 0-16M, or 0-4G, or 0-64G 

PCI defines addressable memory as 0 to 4G and 

everything above 4G 

l 

PCI I/O memory space must be located between 0-4GB 

– Whether this is memory mapped or uses special I/O based 

instruction (i.e., IN or OUT instruction in conjunction with 

hardware controlled I/ORD# and I/OWR#) is not specified by 

PCI 

PCI configuration space is a separate, dedicated 

configuration memory area on each PCI device 

for each function within a device 

l 

This is not related to host memory or I/O address space 


1-62

PCI Configuration Header Types 

n 

First 16 DWords of a PCI function’s configuration 

memory is called the header space 

n 

Three header formats are currently defined 

l 

l 

l 

Header type zero 

– All PCI devices other than PCI-to-PCI bridges 

Header type one 

– PCI-to-PCI bridges 

Header type two 

– Cardbus bridges (not covered in this class) 

n 

Each PCI function on a device has its own 

configuration space 


1-63

PCI Configuration Address Space 

DWord 

63 

Configuration 

Space (optional) 

Conventional PCI 


16 

15 

00 

Configuration 

Space (required) 


1-64

PCI Configuration Header Type 0 

PCI Configuration Space 

Device ID 

Status Register 

Class Code 

Vendor ID 

Command Register 

Device ID 

BIST Header Type Latency Timer Cache Line Size 

Subsystem ID 

Base Address 0 






CardBus CIS Pointer 

Expansion ROM Base Address 

Reserved 

Reserved 

Subsystem ID Vendor 

Capabilities 

Pointer 

Max_Lat Min_Gnt Interrupt Pin Interrupt Line 

Dword 

00 

01 

02 

03 

04 

05 

06 

07 

08 

09 

10 

11 

12 

13 

14 

15 

Shaded registers are required on all PCI devices, 

registers not shaded are optional 


1-65

PCI Configuration Header Type 1 (PCI Bridge) 

Device ID 

Status Register 

Class Code 

BIST Header Type Latency Timer 

Secondary 

Latency Timer 

Secondary Status 

Memory Limit 

Prefetchable 

Memory Limit 

I/O Limit 

Upper 32 Bits 

Bridge Control 

Base Address Register 0 

Base Address Register 1 

Subordinate 

Bus Number 

Expansion ROM Base Address 

Vendor ID 

Command Register 

I/O Limit 

Prefetchable Base 

Upper 32 Bits 

Prefetchable Limit 

Upper 32 Bits 

Reserved 

Secondary 

Bus Number 

Interrupt Pin 

Memory Base 

Prefetchable 

Memory Base 

Revision ID 

Cache Line 

Size 

Primary 

Bus Number 

I/O Limit 

I/O Base 

Upper 32 Bits 

Capabilities 

Pointer 

Interrupt Line 

Dword 

00 

01 

02 

03 

04 

05 

06 

07 

08 

09 

10 

11 

12 

13 

14 

15 


1-66

Host PCI Bridge Discovery 

n 

n 

n 

Access to a PCI bus is by a host/PCI bus bridge 

and is always referred to as PCI bus 0 

– There can be several host/PCI bus bridges 

– Each host/PCI bus bridge will be uniquely numbered 

• How each bridge is numbered/identified will be system specific 

and dependent on the host processor(s) and local OS 

A PCI/PCI bridge or P-P bridge may provide 

access to other PCI busses 

l 

PCI devices reside on either a primary or a secondary PCI 

bus 

– A primary bus is the bus closer to the host/PCI bridge or 

upstream PCI bus 

– A secondary bus is the bus further away from the host 

processor or downstream PCI bus 

Configuration Reads/Writes used to discover 

busses and devices from host/PCI bus bridge 


1-67

PCI/PCI-X Primary/Secondary Bus Scheme 

Processor Processor Processor 

Bus=0 

Subor=2 

Host/PCI 

Bridge 1 

Main 

Memory 

Host/PCI-X 

Bridge 

Bus=3 

Subor=4 

PCI Bus 0 

PCI/PCI 

Bridge 

Pri=0 

Sec=1 

Subor=2 

Pri=3 

Sec=4 

Subor=4 

PCI-X/PCI-X 

Bridge 

PCI Bus 3 

Pri=1 

Sec=2 

Subor=2 

PCI/PCI 

Bridge 

PCI Bus 2 

PCI Bus 1 

PCI Bus 4 

• Primary bus or upstream bus 

• Moves transaction towards Host/PCI bridge 

• Secondary bus or downstream bus 

• moves transaction away from Host/PCI bridge 


1-68

PCIe Primary/Secondary Bus Scheme 



Links are given bus 

numbers during 

configuration 







Upstream Port 


n 

PCIe links are numbered as in PCI – a numerically assigned bus 

hierarchy 

l Each end of a link defines a ‘port’ connection 

l Packets move in and out of ports in upstream or downstream directions 


1-69

Optional Configuration Space 

n 

If the PCI/PCI-X device has optional or enhanced 

capabilities, the capabilities register will contain 

an offset to optional configuration space 

l 

Pointer will be entry point into a predefined structure defining 

the enhanced feature 

Format of Capability register set 

Pointer to 

Next Capability 

Capability ID Dword 0 

Feature specific configuration registers 

(i.e., PCI-X device or PM capable) 

Dword 1 

Dword N 


1-70

Optional Registers (continued) 

Last pointer contains ‘0’ 

Optional Space 

Mandatory Space 

63 

16 

15 

• More than one capability 

register set may be offered 

by a function 

• Capabilities pointer points 

to next register set, with ‘0’ 

indicating the last set in the chain 

0 


1-71

PCIe Configuration Space 

n 

Configuration space on all PCIe devices are 

identical to PCI configuration space 

l 

PCIe devices will have optional PCI and/or PCI-X register 

sets pointed to in optional PCI configuration space 

– Must be PCI PM compliant 

– Must support PCI-X for legacy PCI-X devices 

– Must use MSI if capable of generating interrupts 

– Will have the new PCIe register set 

l 

PCIe devices may optionally implement an extra 3 KB (up to 

1 KW) of configuration space 

– Enhanced error reporting/logging register set and virtual 

channel register sets located in this area 


1-72

PCIe Configuration Space 

DWord 

1000 


Extended Configuration 

Space 

Optional extended PCIe 

configuration space. 

Parameters and capabilities 

defined that are not available on 

existing systems 

Conventional PCI 


64 

15 

0 

PCIe required capability 

structure located in PCI optional 

configuration Space 

PCI 3.0 required 

configuration header 


1-73

Section 2 

PCIe Transaction Layer 



Key Aspects of Transaction Layer 

n 

n 

At a high level, key aspects of 

the Transaction Layer are 

l 

l 

l 

l 

Pipelined full split-transaction 

protocol 

Differentiating TLPs based on 

ordering and processing 

requirements 

Credit-based flow control 

Optional support for data poisoning 

and end-to-end data integrity 

Interaction is to/from a 

Requestor and to/from the 


Requestor 









2-75

Functional Requirements of Transaction Layer 

n 

TLP construction and processing 

l 

l 

Packet construction by originator 

Segmentation by Requestor according to max payload size 

– Reassembly at destination by Completer 

n 

Association of Express transaction-level 

mechanisms with device resources 

l 

l 

l 

Addressing 

Flow control 

Virtual Channel management (Traffic Class differentiation) 

n 

Rules for ordering and management of TLPs 

PCI/PCI-X compatible ordering 


2-76

PCIe Transactions 

n 

Software will target Express transactions 

between system memory, I/O, and device 

configuration operations 

l 

Originator of transaction will deliver command to the 

Transaction Layer of the Express protocol stack 

– Serial conditioning begins by moving the operation 

through the fabric 

l 

The PCIe Requestor/Completer relationship is shown on the 

following slide 

– Requestor/Completer relationship is as defined by the 

PCI-X software model 


l 

Transaction Layer provides end-to-end services 

– Enough information in packet to define transaction type 

from Requestor to Completer 

2-77

Transaction Layer Provides End-to-End Service 

Host System 

Requestor 

(Completer) 

Completer 

(Requestor) 


Application 

Application 

Root Complex 




End-to-End 

Switch 









2-78

Address Space and Transaction Type 

n 

PCIe defines four address areas or spaces, and 

several transaction types 

l 

Each transaction type has its own unique intended usage 

– Construction of TLP headers can vary depending on 

transaction type 

Address Space 

Memory 

I/O 

Configuration 

Message 

Transaction Types 

Read 

Write 

Read 

Write 

Read 

Write 

Baseline 

Vendor-Defined 

Transaction Usage 

Transfer data to/from a 

memory-mapped location 

Transfer data to/from an 

I/O-mapped location 

Device configuration/setup 

and control 

From event signaling mechanism 

to general purpose messaging 


2-79

Address Method and TLP Headers 

n 

The following address methods are allowed 

l 

Memory reads and writes 

– Uses 1 or 2 DW defining address (32-bit or 64-bit) 

• Mapped to system memory resources 

l 

I/O reads and writes 

– Uses 1 DW defining I/O address (32-bit I/O addressing 

only) 

• Mapped to system I/O resources (1 DW payload) 

l 

Configuration reads and writes 

– Uses 1 DW defining targeted bus, device, function 

number, and DW location in devices configuration space 


l 

Implicit 

– Destination address defined in the command 

2-80

TLP size Varies 

n 

TLP construction and size depends on the 

command and the addressing method 

DW 

0 1 2 3 4 5 4134 4135 

TLP Header 

TLP Header 

TLP (3 DW header only) 

TLP (4 DW header only) 

TLP Header 

ECRC 

TLP (3 DW header w/ ECRC) 

TLP Header ECRC TLP (4 DW header w/ ECRC) 

TLP Header 

Data 

TLP (3 DW header w/data) 

TLP Header 

Data (4 KB max) 

ECRC 

TLP (3 DW header w/data 

and ECRC) 

TLP Header 

Data 

TLP (4 DW header w/data) 

TLP Header 

Data (4 KB max) 

ECRC 

TLP (4 DW header w/data 

and ECRC) 


2-81

TLPs and Transactions 

n 

Transactions are carried using Requests and 

Completions 

l 

Allowable combinations of Read/Write Request and 

Completion operations vary depending on transaction type 

n 

TLP headers and TLP packet size are variable 

l 

TLP headers can be 3 or 4 DW in length 

– TLP packet may optionally have a 32-bit ECRC 

appended to the packet 

– A TLP packet may or may not carry a data payload 

• 4 KB maximum data payload, 1 DW minimum payload 


2-82

PCIe Transaction Movement 

Host System 





Memory R/W 

Requestor 

Completer 

Memory R/W 

I/O R/W 

I/O R/W 

Messages 

Messages 

Root Complex 







Transaction Layer Packet 


Express Packet 


2-83

TLP Packet Components 

n 

DWORD 

DW 0 

DW 1 

DW 2 

Packets are usually illustrated as a stacked 

sequence of DWs (4 bytes) 

l 

Packet length is always in DW increments (TLP with 3 DW 

header shown) 

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 

Data Byte 0 

BYTE 

+0 +1 +2 +3 

TLP Header 

Data Payload 

DW n-1 

DW n 

TLP Digest (optional) 

Data Byte (N-1) 


2-84

TLP Packet Components 

n 

TLP packets are sent in bit-wise serial format as 

shown below 

l 

Three distinct parts of a TLP packet 

– TLP Header 

• Defines how packet will be constructed and processed by 

source and destination (and any fabric devices in 

between) 

– Data Payload (when applicable) 

– An optional TLP Digest (ECRC) 

TLP Header 

Data Payload 

(OPTIONAL) 

TLP Digest 

(OPTIONAL) 

DWord 0 

1 2 N-1 

N 

TLP Packet 


2-85

TLP Transmit Processing 



n 

Command received from application 

l 

TLPs created as specified by the command 



DWord 

TLP Header 

0 1 2 

Data Payload 

N-1 

TLP Digest 

(optional) 

N 

TLP 

Packet 




TX RX 

n 

Passed to Data Link Layer for further 

processing 


2-86

TLP Receiver Processing 

n 

Received by Data Link layer, 

analyzed, and processed 

l 

If appropriate, data or command passed 

to application layer 




TLP Header 

X 

Data Payload 

To Application Layer 

Data Payload 

DWord 0 1 2 

N-1 

TLP Packet 

TLP Digest 

(optional) 

X 

N 





From Data Link Layer 

TX 

RX 


2-87

Common Part of TLP Header 

n 

All PCIe TLPs contain the following common TLP 

header fields 

l 

The Fmt and Type fields determine the size and 

interpretation of the remaining part of the header 

+0 +1 +2 +3 

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 

R Fmt Type R TC R 

T 

D E P 

Attr AT Length 

TLP Header 

Data Payload 

TLP Digest 

Byte 

0 1 2 

3 

Remainder of TLP Header 


2-88

Message Codes 

n 

Messages are defined for 7 groups (Message 

Code Field) 

l 

l 

l 

l 

l 

l 

l 

INTx interrupt Signaling -------- (0010 XXXX) 

Power Management ------------- (0001 XXXX) 

Error Signaling -------------------- (0011 XXXX) 

Locked Transaction Support -- (0000 XXXX) 

Slot Power Limit Support ------- (0101 XXXX) 

Vendor-Defined messages ---- (0111 XXXX) 

Hot-Plug Signaling --------------- (0100 XXXX) 


2-89

PCIe Flow Control 

n 

Two types of Flow Control (FC) are provided 

l 

l 

At the Transaction layer 

– Used to prevent overflow of Receiver buffers and enable 

compliance with ordering rules 

– Used by Requestor to track queue/buffer space 

available in the Agent across the Link and control TLP 

injection rate 

– The Transaction Layer performs FC accounting 

functions for Received TLPs and ‘gates’ TLP 

transmissions based on available ‘credits’ for 

transmission 

At the Data Link Layer 

– Exchanges ‘credits’ on the Link in a Data Link Layer 

Packet (DLLP) between Transaction Layers at each end 

of the link 

• This form of FC is transparent to the Transaction Layer 


2-90

Section 3 

PCIe Data Link Layer 



Data Link Layer Provides Point to Point Service 

Host System 


Requestor 

Completer 

Application 

Application 

Root Complex 




End-to-End 

Switch 




Point-to-Point 







3-92

Key Aspects of Data Link Layer 

n 

At a high level, key aspects of the 

Data Link Layer are 

l 

l 

Accept TLPs from the Transaction 

Layer, provide Data Link processing, 

and pass to Physical Layer 

Accept TLPs from the Physical Layer, 

process Data Link Layer header, and, if 

appropriate, pass to the Transaction 

Layer 





n 

Data Link Layer is responsible for 

Link initialization, Link 

maintenance, and data integrity 

on a link-by-link basis 





3-93

Transmit Side of Data Link Layer 

n 

Two packets types are created and 

processed at the Data Link Layer 

l 

l 

Conditioned TLPs 

– TLPs received from the 


• Pre-pended with a 16-bit Sequence 

Number 

• Appended with a 32-bit Data Link Layer 

CRC (LCRC) 

Data Link Layer created packets (DLLPs) 

– TLP Ack/Nak packets 

• Ack/Nak for receive side 

– Link management packets 

• Link training, initialization, and PM 

management 

• Flow Control (FC) packets 









3-94

Data Link Layer Created Packets - Transmit 

Transaction Layer block 

passed to Data Link Layer 


DLLP created solely by 

and at the Data Link Layer; 

link management packets 


Data Link Layer ‘conditions’ 

block to create a TLP packet 




Physical Link 


3-95

Receive Side of Data Link Layer 

n 

Data Link Layer receiver 

l 

Processes received TLPs 

– Validates Sequence number, 

computes and compares LCRC 

• Strips fields and passes to Transaction 

Layer 



l 

Processes received DLLPs 

– Processing of Ack/Nak, PM, and 

FC DLLPs is done in hardware 

• May or may not have information passed to 

Transaction Layer as a result of DLLP 

processing 




Physical Link 


3-96

Data Link Layer Received Packets 

Passed to Transaction 

Layer receiver 


Link management 

DLLPs processed and destroyed 





Data Link Layer validates 

Received TLP Sequence 

Number and verifies LCRC 

Physical Link 


3-97

Functional Requirements of Data Link Layer 

n 

The Data Link Layer comprehends the following 

l 

DLLP construction 

– Packets for Link services 

• Link initialization, training, link power management 

– Packets for TLP acknowledgements 

• All received TLPs generate an Ack or Nak DLLP 

– ID routed to Transaction ID from the received TLP 

n 

Data Link Layer provides conditioning and 

tracking of TLPs for normal operation 

n 

DLLPs and TLPs are the primary packets seen on 

the Link 


3-98

Data Link Layer Handling of TLPs 

n 

Data Link Layer stores a copy of every transmitted 

TLP until receipt has been acknowledged 

l Purges copies only when acknowledgement (Ack) has been 

received 

– Received Nak forces re-transmission 

– Data Link Layer timer roll over also causes retransmission 

l 

Memory buffer and control for this storage will be vendor 

specific 


3-99

Data Link Layer Control 

n 

The Data Link Layer may be in any of three states 

and provides two status indicator outputs 

l Data Link Layer states 

– DL_Inactive 

• Link is non-operational or nothing is happening 

– DL_Init 

• Link is operational 

• Flow control initialized for default VC (VC0) 

– DL_Active 

• Normal operational mode 

l Data Link Layer status outputs 

– DL_Down 

• Link is not communicating with component at the other end 

– DL_Up 

• Communications established with component at the other end 


3-100

Link Initialization and Flow Control Protocol 

n 

The first VC initialized must be VC0 and set up for 

the FC default 

l 

Each subsequent VC must be initialized in the same manner 

before being enabled 

n 

VC initialization is triggered by 

l 

Power-on Reset, software generated Fundamental Reset, 

Link re-training, or when initially attached 


3-101

Initialization - TS1 and TS2 Ordered Sets 

Switch 

TS1 TS1 TS1 TS1 

TS1 

TS1 

TS1 

Bit and symbol lock, link width 

determination, and lane-to-lane de-skew 

Endpoint 

TS1 

TS2 TS2 TS2 

TS2 

Switch 

TS2 

TS1 

TS1 

TS1 

TS1 

Endpoint 

Link established with least common denominator between two points determined. 

Data Link layer now starts sending FC_INITF1 sequences to establish flow control. 


3-102

Initialization 

n 

Example TS1/TS2 

l 

LeCroy PETracer 


3-103

Flow Control - Phase of Initialization 

n 

FC_INIT1 

l 

l 

Initial transmission started by endpoints 

– Repeated sequence of three FC_INIT1 DLLPs 

• InitFC1 – P (posted Requests) 

• InitFC1 – NP (non-posted Requests) 

• InitFC1 – Cpl (associated with non-posted Requests) 

– Process all incoming DLLPs for FC_INIT1, INIT2 

– Sequence repeated at wire speed until complete 

sequence returned 

• Set FC_INIT1 P, NP, and Cpl flags set as FC_INIT1 DLLPs are 

received; check for all flags set (FI1 = 1) 

– When FI1 flag is set enter FC_INIT2 phase 

– Set FC_INIT2 P, NP, and CLP flags as FC_INIT 2 DLLPs are 

received, setting F2 flag when all are received 

• Exit when F2 flag set 

Link is now operational 


3-104

Flow Control 

n 

Example credit exchange 

l 

LeCroy PETracer 


3-105

Data Link Layer Created Packets 

n 

The Data Link Layer creates and processes two 

types of packets 

l 

DLLPs 

– Used for Link management and control 

l 

TLPs 

– Adds header (sequence number) and LCRC to 

TLP received from Transaction Layer 

• LCRC provides data integrity for the TLP between two ports of a 

Link 


3-106

Basic DLLP Header Fields 

n 

All Data Link Layer service DLLPs include the 

following fields 

l 

l 

Type – identifies the interpretation of the DLLP 

A 16-bit CRC 

– Not the same as ECRC or LCRC 

– Only applies to DLLPs, not TLPs 

Format depends on 

DLLP type 

+0 +1 +2 +3 

7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 7 6 5 4 3 2 1 0 

DLLP type 

16-bit CRC 


3-107

n 

n 

Data Link Layer Services 

PCIe defines the following DLLP types relating to Link 

services 

l 

l 

l 

l 

Ack/Nak of DLLPs received from the Physical Layer 

Initialization and Flow Control Update 

Power Management (PM) 

Vendor Specific 

Services provided to Transaction Layer 

l 

l 

Receive TLP from Transaction Layer 

– Pre-pending a sequence number to the TLP 

– Calculating and adding a 32 bit Link CRC (LCRC) to 

the TLP 

• Sending to Physical Layer 

Validates received packets sent to Transaction Layer 

– Ack/Nak as appropriate 

– Error log/generation as appropriate 


3-108

Transaction Layer Buffer Space and Credits 

n 

Credits reported by the Data Link Layer indicate 

buffer space available for receiving at the other 

end of the link 

Device A 

Device B 



PH 

PD 



1 Flow Control credit 

received here 

2. Indicate buffer 

space available 

here 








3-109

PCIe Link PM State Diagram 

L0s 

L0 

L2 

L1 

L2/L3 

Ready 

L3 


3-110

PCIe Power Management Overview 

n 

PCIe PM provides the following services 

l 

l 

l 

l 

Mechanism to identify power management capabilities of a 

given function 

Ability to transition a function into a certain PM state 

Notification of the current PM state of a function 

The option to wakeup the system on a specific event 

n 

PCIe PM is compatible with the PCI PM 

l PCI Bus Power Management Interface Specification Revision 

1.1 and the Advanced Configuration and Power Interface (ACPI) 

Specification revision 2.0 

– Legacy PM software will have to be re-written to 

take advantage of PCIe PM enhanced capabilities 

• PCIe PM states are not directly visible to legacy bus driver software 


3-111

PCIe Power Management Overview 

n 

n 

PCIe PM involves transitioning devices through L 

states 

l PCIe Active State Power Management (ASPM) allows 

autonomous hardware based active state control of a device 

– If the Link is inactive for a period of time, 

ASPM transitions the device to a lower power 

state 

PCI PM involves ‘D’ states 

l 

Express devices emulate D state transitions by moving the 

device through Express L states 

n 

The Link Training and Status State Machine 

(LTSSM) controls device’s Data Link and 

Physical Layer operation 


3-112

Link Training and Status State Machine (LTSSM) 

L states represent 

Express power level 

transitions 

Detect 

Polling 

Disabled 

Configuration 

Hot Reset 

L2 

L0 

Loopback 

L1 

L0s 

Recovery 


3-113

Link Training and Status State Machine (LTSSM) 


3-114

Normal Operations 

n 

Once the Link is initialized and operational, the 

Data Link Layer conditions TLPs received from 

the Transport Layer and passes them to the 


l Data Link Layer service DLLPs will show up as traffic 

intermittently 

– Exchange of flow control credits, Ack/Nak, 

power transition commands 

n 

Data Link Layer TLP conditioning consists of 

l 

l 

Pre-pending a Sequence Number to the TLP and 

appending a 32-bit link layer CRC (LCRC) 

Validating received LCRCs, acknowledgement, and passing 

TLP to Transaction Layer on the receiver side 


3-115

Normal Data Link Layer Operation 

n 

PCIe describes Sequence Number generation in 

terms of conceptual counters, timers, and flags 

l 

Actual silicon implementation (registers and field bit 

definitions) of these functions will be vendor specific 

n 

Normal operation involves passing conditioned 

TLPs to the Physical Layer and receiving an Ack 

or Nak referencing a specific TLP 

l 

Flow control, buffer sizes, and link utilization will be 

determined by use of these counters, timers, and flags 

n 

Implemented on a link-by-link basis throughout 

the fabric 


3-116

Sequence Number - Transmit 

Requestor 


n 

NEXT_TRANSMIT_SEQ counter number 

appended to TLP (plus 4 reserved bits) 

l Incremented for each successive TLP sent 

Seq1 

Seq2 

Seq1 

TLP 

TLP 

TLP LCRC 

Seq2 TLP 

TLP LCRC 

TLP LCRC 

LCRC 

Data Link 

Layer 

n 

ACKD_SEQ counter represents latest 

Ackor Nakreply 

l Copy of TLP Seq’X’ retained until Ack 

received 

l If (NEXT_TRANSMIT_SEQ – ACKD_SEQ) 

mod 4096 >= 2048 then the Data Link Layer 

transmitter must stop accepting TLPs from 

the Transaction Layer 


Ack1 DLLP 

NEXT_TRANSMIT_SEQ 

Counter 

ACKD_SEQ 

Counter 

TLP Seq1 


3-117

Priority of Scheduled Transmission 

n 

PCIe lists the following order of priority for 

transmission of TLPs and DLLPs (this is 

recommended and is not mandatory) 

l 

l 

l 

l 

l 

l 

Completion of any TLP or DLLP currently in progress 

Nak DLLP transmissions 

Ack DLLP transmissions 

– Scheduled as soon as possible after receipt of 

a duplicate TLP or expiration of the Ack 

latency timer 

FC DLLP transmissions 

– Flow Control credit exchange 

Retry Buffer re-transmissions 

TLPs from the Transaction Layer 


3-118

Section 4 

PCIe Physical Layer 



Physical Layer Transmits/Receives 1s and 0s 

Host System 


Requestor 

Completer 

Application 

Application 

Root Complex 




End-to-End 

Switch 





Bits 





Bits 


10010101100 10110101101 


4-120

Physical Layer Overview 

n 

The PCIe Physical Layer is 

composed of two sub-blocks 

that work in concert 

l 

l 

Logical sub-block 

Electrical sub-block 



n 

A second volume (PCIe Card 

Electromechanical 

Specification Revision 1.0a) 

details connector types and 

associated electrical and 

mechanical requirements 


Logical Sub-Block 

Electrical Sub-Block 

Fabric Connection 


4-121

Sub-Block Interaction 

n 

Logical and Electrical sub-blocks each have two main 

sections 

l Transmit section 

– Logical sub-block prepares outgoing packets received 

from the Data Link Layer, performs 8B/10B symbol 

encoding, and passes symbols to the Electrical subblock 

– Electrical sub-block takes conditioned symbols received 

from Logical sub-block and transmits in a serial bit 

stream out the device’s physical transmit port 

l Receive section 

– Electrical sub-block receives a serial bit stream over the 

device’s physical receive port, recovers the clock, strips 

the framing bits, and passes a symbols to the Logical 

sub-block 

– Logical sub-block performs 10B/8B decoding and 

passes packet to the Data Link Layer 


4-122

Physical Layer Packets 

Packet delivered to 


Block passed from 

Data Link Layer to 






Encoded and framing 

Symbols added 


Framing and packet 

decoded, framing 

stripped 

Physical Link 

1 

1 

0 

0 

0 

0 

1 

1 

0 

1 


4-123

Electrical Layer Block Diagram 

8 Bit 

Logical Sub-Block 

8 Bit 

10 Bit symbols 

8B/10B Encoding 

Framing symbols 

added 

10B/8B Decoding 

Framing symbols 

stripped 

10 Bit symbols 

Parallel 

To 

Serial 

Converter 

Serial 

To 

Parallel 

Converter 

Bits 

1 

0 

0 

1 

Serial Bit Stream 

to 1.2 V Differential 

Signal 

Electrical Sub-Block 

1.2 V Differential 

Signal to 

Serial Bit stream 

Bits 

1 

0 

0 

1 

Tx 

Rx 

x1 Physical Lane 

(card electrical and mechanical 

specification) 


4-124

Scrambling 

n 

Scrambling is a technique for rearranging the 1s 

and 0s for any given 8 bit/10 bit conversion 

n 

Preformed in hardware prior to 8B/10B conversion by 

the transmitter and after 10B/8B conversion by the 

receiver 

n 

n 

Default is enabled 

Used to consecutive 1s or 0s transmitted to reduce 

power levels/cross-talk during transmission 

n 

If used on a multi-lane link same hardware will 

algorithm will be used on all lanes 

n 

Framing characters will not be scrambled 


4-125

Logical Sub-Block Services 

n 

The Physical Layer Logical sub-block performs the 

following 

l Symbol encoding/decoding 

– Per ANSI X3.230-1994, clause 11, and IEEE 802.3z, 

36.2.4 

• Encoding allows for 12 special symbols used for framing and Link 

management 

• Used for Link Transmitting 

– Adding framing symbols to symbol encoded packet, feeding packet 

byte stream to the appropriate Physical Layer Lane 

• Used for Receiving 

– Receiving serial byte stream from device’s receive port, stripping 

framing symbols, decoding received packet, and passing packet to 

the Data Link Layer 

l 

Link initialization, training, and status state rules 

l 

Link error recovery 


4-126

8B/10B Signal Encoding 

Data Byte 

Name 

Data Byte 

Value 

Current Current 

Bits 

RD- 

RD+ 

HGF EDCBA abcdei fghj abcdei fghj 

D0.0 

00 

000 00000 100111 0100 

011000 1011 

D31.7 

FF 

111 11111 101011 0001 010100 1110 

Byte stream 

8 bits 

3 bits 5 bits 

4 bits 6 bits 

Symbol stream 

10 bits 

To physical transmitter 


4-127

Encoded Packets 

“D” Characters 

STP 

Sequence # 

Transaction Layer Packet (TLP) 

LCRC 

END 

“K” Characters 

(framing) 

“D” Characters 

SDP 

Data Link Layer Packet (DLLP) 

END 

“K” Characters 

(framing) 


4-128

Special Symbols 

n 

Framing and Link management use special ‘K’ 

coded symbols 

l 

l 

l 

l 

l 

l 

l 

l 

l 

COM (comma) – used for Lane and Link initialization and 

management 

STP (start TLP) – marks the start of a TLP packet 

SDP (start DLLP) – marks the start of a DLLP packet 

END (end) – marks the end of a TLP or DLLP packet 

EDB (end bad) – marks the end of a nullified TLP 

PAD (pad) – used in framing, Link Width, and Lane ordering 

negotiations 

SKP (skip) – used for compensating for different bit rates for 

two communicating ports 

FTS (fast training sequence) – used within an ordered set to 

exit from L0s to L0 

IDL (idle) – used in electrical ordered set 


4-129

TS1 and TS2 Ordered Sets 

Symbol 

Number 

0 

1 

2 

3 

4 

5 

6 

13 

14 

15 

Symbol 

COM 

Link# 

Lane# 

N_TSF 

Rate ID 

Train Ctl 

TS ID 

TS ID 

TS ID 

K28.5 

D0.0-D31.0, K23.7 (0-255) 

D0.0-D31.0, K23.7 (0-31) 

Number of FTSs required by receiver to obtain bit and symbol lock 

D2.0 = 2.5 Gbps 

Training Control 

D10.2 for TS1, D5.2 for TS2 



Bit 0 0 = De-assert Hot Reset 

1 = Assert Hot Reset 

Bit 1 0 = De-assert Disable Link 

1 = Assert Disable Link 

Bit 2 0 = De-assert Loop back 

1 = Assert Loop back 

Bit 3 0 = De-assert Disable Scrambling 

1 = Assert Disable Scrambling 

Bits Reserved 

4-7 


4-130

Other Ordered Sets 

n 

n 

Electrical Idle 

l 

l 

Skip 

l 

Transmitted to a receiver prior to the transmitter placing its 

transmit half in electrical idle condition 

COM symbol followed by three IDL symbols 

– After transmitting this ordered set, transmitter 

drives a differential of less than 20 mV peakto-peak 

COM symbol followed by three SKP symbols 

– Used for clock tolerance compensation 

– Must be scheduled for insertion once every 

1180 symbols 

– Transmitted simultaneously on all lanes of a 

link if a multi-lane link connection 


4-131

Link Initialization 

n 

Link Initialization consists of the following 

l Configuring 

– Discovering and determining Link width, data rate, lane 

reversal, and polarity inversion 

– Restarting a port from low power states 

l Link training 

– Data rate negotiation 

– Bit lock, symbol lock, and lane priority on a per lane 

basis 

– Link width negotiation 

n 

Receivers may optionally check for violations of the Link 

Initialization and Training protocols 

l If enabled to do so and a training error occurs it is reported as a 

Training error with the associated port 

– A Training error is considered to be fatal to the Link 


4-132

Link Training Ordered Sets 

n 

Using training ordered sets TS1 and TS2, each end of the 

link determines the following 

l Bit lock 

– frequent 1 0 transitions allow receiver to ‘bit lock’ onto 

the transmitters clock 

– Link signaling rate 

• 2.5 Gbps current support (enhancements planed) 

l Symbol lock 

– Uses COMM symbol as the start of TS1 and TS2 

training sets 

• Establishes symbol boundary sensing at receiver 

l Lane-to-Lane de-skew 

– Receiver adding/removing fixed latency on each lane of 

a link to align serial bit stream of the packet across the 

link 

• Compensates for trace and impedance mismatches due to routing, board 

materials, and within transceivers 


4-133

End 

This presentation just scratches the surface… 

Learn more http://dashcourses.com/courses/pcie/pcie-fundamentals.html 

As for the PLX discount.

PCIe Devices - PLX Technology

Create successful ePaper yourself

Delete template?

Save as template?