DesignCon 2002
DesignCon 2002
DesignCon 2002
Create successful ePaper yourself
Turn your PDF publications into a flip-book with our unique Google optimized e-Paper software.
<strong>DesignCon</strong> 2008<br />
BladeServer 10 Gb/s Ethernet<br />
Backplane Design With<br />
Equalization Using Statistical<br />
Channel Analysis<br />
Pravin Patel, IBM<br />
Email: pravinp@us.ibm.com<br />
Phone: +1-919-543-3837<br />
Jeff Cutcher, Ansoft Corporation<br />
Email: jcutcher@ansoft.com<br />
Tony Donisi, Ansoft Corporation<br />
Email: tony_dw@comcast.net<br />
Michael Tsuk, Ansoft Corporation<br />
Email: mtsuk@ansoft.com<br />
Steve Pytel, Ansoft Corporation<br />
Email: spytel@ansoft.com
Abstract<br />
A fundamentally important aspect of high speed channel design at 10 Gb/s is that<br />
significant optimization of the physical channel is required. Path degradation in channels,<br />
from ISI effects, causes poor signal quality and must be minimized or eliminated. This<br />
paper explores how equalization, statistical analysis, and accurate models can be used to<br />
optimize high speed channels. A case study, based on a 10 Gb/s BladeServer Ethernet<br />
backplane, will serve to illustrate how an integrated approach using equalization, accurate<br />
physical models, and statistical simulation can be used to optimize the design of an<br />
industrial application.<br />
Author(s) Biography<br />
Pravin Patel<br />
Pravin Patel pravinp@us.ibm.com Pravin Patel is a Senior Engineer and Technical<br />
Leader working for IBM xSeries Server Development. He is involved in defining product<br />
architecture, creating product strategies, and time and Frequency Domain SI<br />
analysis/simulation of Intel base server products. His current activities include the design<br />
and analysis of Serdes interfaces for BladeCenter product. He performs modeling,<br />
simulation and measurements of the high speed serial link Channels. The other areas of<br />
his responsibilities include design and development of models for line cards, backplanes,<br />
traces, via and performing simulations for system level voltage and timing budgets and<br />
jitter characterization. He is an active member of T11.2 and IEEE802.3 channel model<br />
standard committee from IBM. He received a B.S. degree in electrical engineering from<br />
the New Jersey Institute of Technology in 1989. Pravin has received 12 U.S. Patents, 3<br />
IBM Invention Plateau Achievement Awards, 7th IBM Authors Recognition Awards, one<br />
IBM Outstanding Technical Achievement Awards, and One IBM Corporate Award.<br />
IBM. He has authored or co-authored 25 technical papers in IEEE EPEP, IEEE ECTC<br />
and <strong>DesignCon</strong> conferences.<br />
Jeff Cutcher<br />
Jeff Cutcher joined Ansoft Corporation as an Applications Engineer in 2005 with a focus<br />
on supporting their Designer, Nexxim, and SIwave products. Prior to joining<br />
Ansoft, Jeff held the position of Senior Staff Electrical Engineer at Motorola, where he<br />
was involved in various wireless communications development projects that included<br />
digital, analog, and RF circuit design, to pcb layout, and system modeling and simulation.<br />
Jeff is currently a Senior Member of the IEEE, a registered Professional Engineer in the<br />
State of Florida, and holds his BSEE and MSEE from New Jersey Institute of<br />
Technology.<br />
Tony Donisi<br />
Tony Donisi has been a Senior Applications Engineer for Ansoft since 2000. During his<br />
time with Ansoft, he has developed a broad range of applications and designs in many<br />
different areas, such as RF & microwave circuits, wireless communication systems, and<br />
SI related products. Prior to Ansoft, he worked as a RF design engineer for over 17
years. He worked at companies including Raytheon, M/A-Com, and Alpha industries.<br />
He was primarily involved with military and commercial receiver and transmitter module<br />
design, including MIC’s and MMIC’s. Design and analysis work included broadband<br />
and low noise MMIC & MIC amplifiers, components and subsystems; image reject and<br />
sub-harmonic mixers; frequency conversion modules; phase locked loops; monopulse<br />
radar systems and microwave and millimeter wave T/R modules. He earned his BSEE<br />
from Northeastern University, Boston, MA and his MSEE from UMASS Amherst.<br />
Michael Tsuk<br />
Michael Tsuk received the BSc, MSc, and Ph.D. degrees in Electrical Engineering from<br />
MIT in 1984, 1986, and 1990, respectively. In 1990, he joined Digital Equipment<br />
Corporation, where his primary responsibilities were designing and maintaining Digital’s<br />
internal electromagnetic CAD software, and consulting on its use for signal integrity,<br />
power integrity, and EMI modeling. He was promoted to Consulting Engineer at Digital,<br />
became a Member of the Technical Staff after Digital’s merger with Compaq in 1998,<br />
and an Expert after Compaq’s merger with Hewlett-Packard in <strong>2002</strong>, after each merger<br />
expanding the user base of his software throughout the larger corporations.<br />
In 2003, he joined Ansoft Corporation, becoming part of the Nexxim team,<br />
where his primary responsibilities concern models for transient simulation of lossy<br />
coupled transmission lines and general frequency-dependent structures described by<br />
scattering parameter matrices.<br />
He is the holder of four patents and is the author of numerous technical papers.<br />
Steve Pytel<br />
Steve Pytel joined Ansoft Corporation as an Applications Engineer in 2007 focusing in<br />
high speed interconnects; he supports tools such as Ansoft Designer, Nexxim,<br />
SIwave, HFSS, and Q3D. Prior to joining Ansoft, Steve held the position of a<br />
Senior Hardware Design and Signal Integrity Engineer at Intel Corporation where he was<br />
responsible for the signal integrity of Telco, Enterprise, and ATCA Blade form factor<br />
servers. He also focused on high speed electrical interconnect modeling research where<br />
he and a team developed frequency dependent dielectric transmission line models that<br />
incorporated various forms of copper surface roughness loss. Steve received a Bachelors<br />
of Science in Electrical Engineering from Northern Illinois University, a Masters of<br />
Engineering, and a Doctor of Philosophy in Electrical Engineering from the University of<br />
South Carolina.
Introduction<br />
High-speed channel designs, such as 10 Gb/s Ethernet, PCIe 3.0, 8.5 Gb/s Fiber channel,<br />
demand accurate channel simulation and modeling to ensure first-pass system success in<br />
a BladeServer environment. The backplane product in a BladeServer is defined here as<br />
the flexible interface to attach shared devices such as processor blades, 10 Gb/s Ethernet<br />
switches, management modules, etc… Modeling was used, primarily, to guide decisions<br />
on the design trade-offs which promote system-level compatibility between shared<br />
devices. The simulation strategy was developed to balance implied business goals (cost,<br />
time-to-market) against the requirements of product performance. This paper discusses<br />
the implementation of simulation to cope with AC loss (the predominant concern), with<br />
inter-symbol interference (ISI) effects due to impedance geometry changes such as via<br />
stubs, and signal jitter. At high data rates, layout discontinuities coupled with long delays<br />
bring on severe ISI that may be mitigated by equalization or topology manipulation.<br />
Statistical methods have been used to rapidly generate two well known channel<br />
performance metrics: eye diagrams and bit error rate (BER) curves. Simulation time<br />
becomes a critical factor because the real-time simulation of standard pseudo-random bit<br />
sequence (PRBS) patterns such as PRBS23 or PRBS31 is impractical for a designer who<br />
is trying to detect BER rates lower than 10 -12 using transient simulators. These statistical<br />
methods provide a useful approximation of channel performance with the benefit of very<br />
fast simulation times.<br />
The BladeServer 10 Gb/s channel design case study incorporates statistical and transient<br />
analyses within a schematic entry simulator. To achieve the greatest channel accuracy,<br />
three-dimensional models of vias and connectors were incorporated along with onedimensional<br />
distributed frequency dependent dielectric transmission line models. The<br />
statistical channel analyses include a combination of Touchstone S-parameter files,<br />
buffers with de-emphasis, w-elements, passive components, feed forward and decision<br />
feedback equalization.<br />
Typical high speed channels include differential traces meandering along some given<br />
length, usually many wavelengths relative to the data rate. The aforementioned statistical<br />
methods offer a quick means of analyzing these high speed channels for impairments<br />
caused by discontinuities. This paper introduces two such methods, one that specifically<br />
analyzes the channel, independent of bit pattern, while the other utilizes a known bit<br />
pattern such as a standard PRBS pattern in order to produce an eye diagram and bit error<br />
rate (BER) plots. The latter analysis produces a BER curve that is as accurate as the<br />
number of bits simulated; however, the statistical nature of the modeling allows for a<br />
quick result unattainable by traditional transient simulation engines. Both analyses will be<br />
explained in detail including the inherent assumptions and modeling details that provide a<br />
fast and accurate simulation result.<br />
Finally, equalization will play a major role in the design of next generation high speed<br />
channels beyond 5 Gb/s. In this paper different equalization schemes will be discussed<br />
including feed-forward equalizers (FFE), continuous time linear equalizers (CTLE), and<br />
decision feedback equalizers (DFE). A multi-tapped FFE combined with a multi-tap DFE
eceiver has been implemented to show corresponding statistical eye information at the<br />
pad of the receiver. Putting it all together; equalization, accurate physical models,<br />
statistical models and methods provide an integrated suite of tools that were utilized in<br />
BladeServer 10Gb/s high-speed channel designs.<br />
The BladeServer Channel<br />
The IBM eServer BladeCenter backplane provides the electrical interconnections among<br />
all of the chassis components, including the processor blades, switch modules,<br />
management modules, media devices, power modules, and blower modules. It also serves<br />
as the redundant power distribution medium from the power supplies to all of these<br />
components. The backplane provides up to 14 blade slots, four switch slots, one media<br />
bay, two management modules, and four power modules, all of which support hot-swap<br />
capability. Interconnections are redundant in order to provide high availability and<br />
maximize uptime for the customer. To provide a multiblade system that supports multiple<br />
protocols, such as Gigabit Ethernet, Fibre Channel, and InfiniBand, over a common<br />
copper-trace medium, a serialized/de-serialized (SerDes) interface was used for this highspeed<br />
internal input/output (I/O) fabric. High-speed designs often require complex, highspeed<br />
printed circuit boards that can easily drive costs higher, usually through the use of<br />
overly stringent design methodologies or expensive printed circuit board materials that<br />
are not required for the system operating parameters. To achieve a cost-competitive<br />
backplane for the BladeCenter system, an extensive pre-design modeling and simulation<br />
analysis was performed to determine the appropriate material and practices to be used in<br />
the design of the backplane. This paper discusses the implementation of solutions to<br />
overcome some of the challenges of 10 Gb/s data transmission over multiple printed<br />
circuit boards and connectors in order to reduce board costs. A benefit of a multi-server<br />
chassis is that it provides the capability to share devices and hence lower the per-server<br />
cost. In the BladeCenter system, all 14 blades share a keyboard and mouse via the<br />
management module, and a Universal Serial Bus (USB) CD-ROM/DVD (compact disc,<br />
read-only memory/digital video disc) drive and USB floppy disk drive (FDD) located in<br />
the front of the chassis. This paper describes the initial pre-design analysis that was<br />
performed to establish the backplane design requirements for both material and design<br />
practices for the high-speed SerDes wiring.<br />
This section describes the electrical design challenges that were associated with the highspeed<br />
SerDes interfaces and were encountered during the definition, design, and<br />
verification of the BladeCenter system. SerDes is a full-duplex, high-speed serial bus<br />
comprising of a single transmit and a single receive differential pair of signals. The<br />
BladeCenter design uses SerDes channels for interconnecting processor blades and highspeed<br />
switches that support Gigabit Ethernet, Fibre Channel, InfiniBand, and Myrinet<br />
I/O fabrics. A comprehensive electrical design methodology, including accurate and<br />
detailed modeling and simulation of the complete design space, was required in order to<br />
achieve the speeds required by these standard interfaces while at the same time using<br />
low-cost printed circuit board material. Some of the obstacles and solutions to supporting<br />
10 Gb/s data transmission over a copper backplane, multiple boards, and multiple<br />
connector technologies are highlighted. The pre-design analysis was performed to predict<br />
interconnect performance for relatively long trace lengths (across three boards and two
connectors) in FR4 material in order to satisfy the high-speed design requirements and<br />
ensure the overall performance objective for this system. The pre-design analysis was<br />
used to create board design guidelines that would prevent signal quality issues and<br />
enhance signal eye opening. The signal and power distribution integrity effects—<br />
including frequency-dependent losses, inter-symbol interference (ISI), crosstalk,<br />
impedance discontinuities, and skew were integrally analyzed across the complete design<br />
space. The signal loss was determined to be the prominent issue. To develop and design<br />
guidelines in a timely fashion before the first hardware prototype was built; the pre-route<br />
simulation capability in Ansoft HFSS, Ansoft Q3D, Ansoft Designer with<br />
Nexxim was used to explore various topology configurations and to determine the<br />
limiting factors for total link loss. Given the system mechanical and electrical boundary<br />
conditions for the high-speed serial link, the topology shown in Figure 1 depicts the<br />
BladeCenter end-to-end SerDes trace design, in which the differential pair traverses three<br />
boards and two connectors with data rates varying from 1 to 10 Gb/s. Simulations were<br />
performed to cover all permutations of design cases to determine the worst-case scenario.<br />
Behavioral device models supplied by external component suppliers and DFE/FFE<br />
equalization techniques were used in fine-tuning the solution space. A variety of tools<br />
and techniques were used to generate a channel model as part of the BladeCenter<br />
backplane pre-design simulation and analysis. Channel models included the backplane<br />
Molex VHDM connectors [1], board traces and vias for the I/O expansion adapter, blade,<br />
backplane, and switch module [2].<br />
The BladeCenter electrical topology, as shown in Figure 1, provides electrical<br />
interconnections between all of the components including the processor blades, switch<br />
modules, management modules and media device. The backplane accepts up to 14 blade<br />
slots, 4 switch slots, 1 media bay, 2 management modules and 4 power modules, all of<br />
which support hot-swap capability. The IBM BladeServer Backplane hardware contains<br />
electrically optimized reference channels constructed using an FR4 loss dielectric with a<br />
½ ounce differential stripline construction using 7 mil trace and 100 ohm impedance<br />
construction. The BladeServer Backplane was chosen to support physical layer testing of<br />
this configuration. The Reference Backplane consists of a backplane test, demonstration<br />
vehicle and a set of line cards to support the operation of up to four adjacent active<br />
channels. Each passive channel contains two VHDM TM connectors, two line cards with<br />
layout optimized connector interface conditions and a passive backplane channel<br />
indicated in Figure 1.
Figure 1: Channel model topology<br />
Each daughter card includes 2.5 to 6 inches of differential signal traces. The remainder of<br />
the channel length is contained within the backplane. The BladeServer Backplane is 24<br />
layers, 225 mils thick, and supports channel lengths from 8 to 32 inches. To minimize<br />
crosstalk, a channel-to-channel separation of 40 mils was determined to provide adequate<br />
isolation throughout the backplane routing. The channel margin hardware contains<br />
selectively degraded-performance channels. These degraded performance channels can<br />
be exchanged with the standard optimized daughter cards in any combination to achieve<br />
discrete parametric control or to achieve the composition of two or more sources of<br />
parametric degradation. Return loss performance can be degraded by a variety of channel<br />
and packaging features such as PCB via stubs formed to accommodate press-fit connector<br />
pins, AC coupling features or PCB layer transition [3].<br />
BladeCenter Packaging<br />
A BladeCenter is a rack-optimized server architecture designed to provide the<br />
consolidation benefits of 1U and 2U rack systems while eliminating the complications<br />
associated with these systems. This is explained in this report by summarizing<br />
BladeCenter package with respect to a 1U rack system package. BladeCenter systems<br />
take less space and put the enterprise server infrastructure within easy reach of the<br />
administrator. BladeCenter provides modular scalability, versatility in server types,<br />
reliability, availability, and serviceability, system management enhancements, thermal<br />
advantage and list of potential cost savings.
The current enterprise infrastructure in the industry is built by 1U and 2U system racks.<br />
Each 1U or 2U server requires its own power cables, Ethernet or Fiber Channel switches,<br />
systems management, power distribution units (PDUs), and keyboard/video/mouse<br />
(KVM) switches. A rack of 42 1U servers can have hundreds of cables strung throughout<br />
the rack, making it difficult to determine where cables are attached and increasing the<br />
complexity of adding or removing servers from the rack. The typical 1U and 2U rack<br />
need to be connected externally to ensure the entire rack is functional.<br />
The BladeCenter unit is a high-density, high-performance rack-mounted server system<br />
developed for medium-to-large businesses. BladeCenter provides up to 14 functionally<br />
separate servers and their shared resources in a single center. The term BladeCenter<br />
package is a chassis that can hold 14 hot swappable devices called blades, which is an<br />
independent server containing one or more processors, memory, disk storage, and<br />
network controllers, 4 hot swappable switches, 2 management modules, 4 power supplies<br />
and 2 blowers. As shown in Figure 2, BladeCenter systems take less space and provide<br />
higher density.<br />
Figure 2: BladeCenter system packaging.<br />
Simulation Methodologies<br />
The following sections describe the methods and results associated with simulating the<br />
high speed SerDes channels. Numerous challenges must be overcome to design a system<br />
that can reliably transmit and receive 2.5 Gb/s serial data over a backplane without errors.<br />
The crucial first step in developing the set of design ground rules to be used in the<br />
development of the BladeCenter backplane was to understand high-speed design criteria<br />
and the effects design materials and methodologies would have. An initial investigation
of high-speed design characteristics yielded four primary areas that had to be taken into<br />
consideration: printed circuit board losses, impedance discontinuities, crosstalk, and skew<br />
due to board routing and connectors. In addition, to validate some of the simulation<br />
results, a number of tests were performed using a standard VHDM backplane evaluation<br />
board to guide decisions for designing and building a backplane capable of supporting a<br />
bit rate of 10 Gb/s. Simulations that used a behavioral channel model were employed to<br />
process all design permutations. The goal was to determine the worst-case scenario and<br />
then optimize the solution space. The mechanical and electrical constraints for the highspeed<br />
serial link, shown in the topology of Figure 1, represent the typical blade design, in<br />
which the signal traverses three boards and two connectors with data rates that vary from<br />
1 to 10 Gb/s.<br />
Accurate modeling of high speed serial channels from the driver to the receiver requires<br />
numerous building blocks that accurately interoperate. Tackling these design issues<br />
would be an impossible task without the use of proper tools such as electromagnetic<br />
models within a circuit simulation environment that can demonstrate silicon performance<br />
including equalization. No longer are purely analytic methods an acceptable solution to<br />
today’s high speed serial design challenges. In addition to pre-layout system simulation<br />
analysis, another key to ensuring proper system operation is post-layout extraction that<br />
quickly analyzes the actual architecture that will be sent off to the fabrication house.<br />
This work will concentrate on a single IEEE 802.3ap KR BladeServer channel that will<br />
include analyses from the driver to the receiver. This BladeServer channel spans over 26<br />
inches of FR-4 and includes three multi-layered circuit boards consisting of a switch card,<br />
backplane, blade card, two high speed connectors, and standard through hole vias (Figure<br />
1). The pre-layout consisted of full wave three-dimensional electromagnetic models<br />
created from Ansoft’s HFSS for the differential vias used within the BladeServer<br />
channel, SMP connectors used for channel measurements, and the VHDM high speed<br />
connectors. The differential traces for the switch, mid-plane, and blade were modeled<br />
using the two-dimensional electrostatic solver within Ansoft’s Q3D. The striplines<br />
were modeled as differential cross-sections that included frequency dependent dielectric<br />
responses of the FR-4 within the PWBs. Ansoft’s Q2D produces a Spice syntax netlist<br />
containing a tabular RLGC w-element file that is used within Ansoft’s Designer with<br />
Nexxim.<br />
This methodology was used to perform pre-layout simulations by defining the channel<br />
within the Designer simulation environment and then performing time-domain and<br />
statistical domain simulations using the Nexxim simulation engine. Initially, a<br />
behaviorally based circuit model with de-emphasis was used to represent the driver side<br />
of the channel. This buffer model consisted of five taps of continuous time linear<br />
equalization.
Figure 3: Blade server simulated channel.<br />
Post-layout modeling was achieved by extracting the Allegro CAD layout for each of the<br />
three PWBs with Ansoft Links and then importing them into Ansoft’s SIwave.<br />
SIwave was then setup to provide a channel model from DC to 20GHz utilizing the 24<br />
layer stack-up for each board. The full-wave, package field solver models each PWB<br />
using a two-dimensional Finite Element Method for all extracted plane geometries and<br />
cavities. This is combined with a Method of Moments solution for each signal trace that<br />
results in a very accurate touchstone or Full Wave Spice file that is used to determine the<br />
system response of the BladeServer. To ensure passive and causal transmission line<br />
responses, the full-wave, package field solver uses a Djordjevic [4] frequency-dependent<br />
model for the dielectrics contained within the PWB.<br />
Finally, time-domain simulations were run for all of the cases so that a prediction of the<br />
system performance at 10 Gb/s could be obtained with and without equalization.<br />
Statistical analysis methods were employed to assist in determining the sensitivity to<br />
transmitter random jitter (TXRJ) and duty-cycle distortion (DCD). Feed forward<br />
equalization and decision feedback equalization algorithms inherent to the circuit<br />
simulator tool were utilized to determine tap weights and eye openings at the pad of the<br />
silicon receiver.<br />
Statistical Analysis – VerifEye<br />
VerifEye is a new, statistical tool for generating contour eye diagrams and bathtub curves<br />
for a channel. By combining the statistics of the bit stream with the variation in
transitions due to jitter, it is possible to generate the information needed in much less<br />
computation time.<br />
First, consider a particular point on a given waveform passed through a channel. That<br />
point is the sum of all the voltage excursions caused by the various transitions from high<br />
to low or low to high at bit interfaces that have occurred prior to the time point in<br />
question.<br />
Rather than waiting for the voltage excursions for all possible combinations of bits and<br />
then looking to see which combination caused an error, the process is shortened through<br />
the calculation of a probability distribution for the sum of all the excursions.<br />
Perhaps it is easiest to consider this concept in terms of what are known as “cursors”; the<br />
deviations from the ideal waveform.<br />
Figure 4: The two largest cursors from the step response of the RC example.<br />
If the channel were perfect, then the response would be identical to the input; the voltage<br />
at each point in time would be either +1V or -1V, depending on the bit in question.<br />
Referring to Figure 4, imagine what the waveform would look like if there were a very<br />
large number of 1s in a row, versus a large number of 0s, followed by a transition to a 1.<br />
In the first case, the waveform would have time to settle to its correct final value of +1V.<br />
In the second case, the waveform would be in transition from -1V to +1V because of
imperfections in the channel. The difference between those two waveforms is a “cursor”,<br />
as shown in Figure 4.<br />
Now, let us explore how we might calculate the inter-symbol interference (ISI)<br />
statistically. Let us assume that the current bit is a 1. This can happen in two different<br />
ways: either the previous bit was a 1, in which case there was no transition between them,<br />
or the previous bit was a 0, in which case there was a low-to-high transition between<br />
them. The probability of a transition is 0.5, if the bits are independent. In the first case,<br />
the cursor is zero; in the second case, it has some significant negative value,<br />
corresponding to the slow rise time of the step response at the end of the channel.<br />
Statistically, therefore, the waveform has a 50% chance of having no deviation, and a<br />
50% chance of a negative deviation, equal to the cursor value.<br />
It is clear that we can chain this process forwards and backwards, considering each<br />
“postcursor” and precursor in turn. The primary difficulty is that, even if the bits are<br />
independent, the transitions are not; a high-to-low transition cannot be followed by<br />
another high-to-low transition; there must be a low-to-high transition in between.<br />
However, this constraint can be handled by the appropriate application of conditional<br />
probabilities.<br />
What comes out of all of the above considerations are probability distribution functions<br />
(PDFs) for the voltage value of the waveforms of high and low bits. These functions can<br />
be combined into cumulative distribution functions (CDFs), contours of which give the<br />
traditional eye diagram.<br />
One of the other measures of the signal integrity of a channel is the so-called “bathtub<br />
curve”. This is a measure of the width of an eye opening at a certain level, usually<br />
halfway between the high and low levels. The bathtub curve turns out to be particularly<br />
easy to generate using statistical methods: it is simply a slice through the CDF<br />
determined above.<br />
Figure 5: Bathtub curve for RC example, with 10ps random transmit jitter. X axis is time in<br />
seconds; Y axis is common logarithm of BER.
One of the great advantages of the statistical technique is that we can easily include the<br />
effects of random transmit jitter (standard deviation and not p-p), without resorting to<br />
Monte Carlo methods. This is done by applying the Gaussian probability distribution<br />
function to the time-location of each cursor, so that the cursors are not just impulses in<br />
voltage, but are spread out due to the Gaussian distribution in time, as shown in Figure 6:<br />
Figure 6: How random transmit jitter smears out the PDFs for the voltage cursors.<br />
One of the advantages of VerifEye over the public-domain tool called StatEye [5] is that<br />
VerifEye uses edges as its basis for calculations, rather than the pulses of StatEye. In<br />
part, this approach is made possible due to the high-quality transient responses that<br />
Nexxim provides for S-parameter-based circuits.<br />
The advantages of edge-based, rather than pulse-based, statistical eye calculations are<br />
primarily in two areas. First, by separating the rising and falling edges of the pulse, we<br />
can ensure that random jitter is indeed truly random: the timing of the edges can be<br />
statistically independent. And also, we can include the effects of duty cycle distortion<br />
(DCD), which can cause the width of pulse to vary.<br />
Pattern-based Analysis – QuickEye<br />
QuickEye uses some simplifying assumptions to create an eye diagram from transient<br />
simulation of a known sequence of single transitions.
A fundamental assumption of both QuickEye and VerifEye is that the drivers and channel<br />
are linear time-invariant systems. QuickEye and VerifEye are based on the principle of<br />
superposition, which makes use of this assumption [6].<br />
The step response is the waveform at the far end of the channel to a single transition at<br />
the input, from low to high or high to low. To obtain the response of the overall channel,<br />
individual step responses for rising and falling edges are superposed. The result is the<br />
channel response shown in Figure 8.<br />
Figure 7: Intermediate QuickEye waveforms for the RC example. Rising edges are shown as solid<br />
curves; falling edges as dashed. When all these transitions are added together, we get the waveform<br />
in Figure 8.<br />
While it is relatively easy to calculate the effects of ISI with the QuickEye algorithm,<br />
random transmit jitter is much more difficult. The most accurate method is to do a<br />
Monte-Carlo method: each transition is displaced a random amount from the nominal<br />
time, depending on the probability distribution of the random jitter, but this greatly<br />
increases simulation time, as there needs to be at least several different jittered transitions<br />
for each bit in the original stream.
Figure 8: The response of a given channel.<br />
Equalization Requirements<br />
Several equalization techniques exist to help combat inter-symbol interference (ISI)<br />
caused by the channel. More often, long channels at high data rates in the Gb/s are<br />
beginning require feed-forward equalization (FFE) followed by a non-linear decision<br />
feedback equalizer (DFE). Typical equalization architectures can be seen in Figure 9.<br />
These examples were directly implemented in the simulation tools. The FFE assists in<br />
sharpening up the edges (pre-cursor) while the DFE utilizes decisions to assist in<br />
removing the interference from previous bits (post-cursor). The computation of the tap<br />
weights is indeed an important step, however, an equally important step is the<br />
determination of the minimum number of taps required to provide an open eye that yields<br />
acceptable BERs. Equalization requirements will be shown for both pre-layout and postlayout<br />
as well as for the measured channel.
Discrete<br />
Time<br />
1.5<br />
1<br />
0.5<br />
0<br />
-0.5<br />
-1<br />
Input<br />
Output (Equalized)<br />
-1.5<br />
-1 -0.8 -0.6 -0.4 -0.2 0 0.2 0.4 0.6 0.8 1<br />
Output<br />
Waveform<br />
Feedback<br />
Taps<br />
K<br />
T T<br />
ff1 ff2 ff3 ffn<br />
fbm fb2<br />
fb1<br />
T<br />
SUM<br />
T<br />
T<br />
T<br />
Decision<br />
Forward<br />
Taps<br />
1<br />
0.8<br />
0.6<br />
0.4<br />
0.2<br />
0<br />
-0.2<br />
-0.4<br />
-0.6<br />
-0.8<br />
-1<br />
20 40 60 80 100 120 140 160<br />
Output<br />
Bits<br />
Figure 9: FFE/DFE Architecture implemented within QuickEye and VerifEye.<br />
The new statistical simulation tools allow the user to determine the number of taps<br />
required for equalizing the channel based on the architecture shown above. The tap<br />
weights may be entered manually or calculated by the software. The input of the<br />
equalizer quantizes the waveform in UI-spaced time slots and passes the voltage<br />
amplitudes through the FFE delays and taps. Currently a zero forcing function has been<br />
implemented to determine the optimum sampling point for the bits along with the tap<br />
weights. At each unit interval (UI) a decision is made between a binary 1 or 0 or (1 and<br />
-1) and that decision is fed back through the feedback delays and taps. The current output<br />
of the equalizer is the sum of both the forward and reverse weighted amplitudes stored in<br />
the UI-spaced taps.<br />
Figure 10: Graphical user interface used for setting up equalization within QuickEye and VerifEye.<br />
Pre-Layout Analysis<br />
Prior to the task of designing and laying out high-speed channels, simulation analysis is<br />
required and performed to determine an acceptable system level performance which is<br />
typically defined by many standards to be a BER = 1x10 -12 at a minimum. Simulation
analysis enables rapid design changes and evaluation without the expense of actual<br />
hardware fabrication resulting in increased profit margin.<br />
The differential strip-line traces were model using a two-dimensional electrostatic<br />
simulation tool with the BladeServer stack-ups being predefined for the switch,<br />
backplane, and blade PWBs. The differential characteristic impedance was defined to be<br />
100 ohms. The electrostatic solver was setup to sweep each stripline cross-section from<br />
DC to 20GHz and a distributed RLGC tabular model was generated for each board. The<br />
2D electrostatic field solver incorporates a Djordjevic model to capture passive, causal<br />
transmission line responses The Djordjevic dielectric model uses a seed parameter<br />
specified at a specific frequency point for loss tangent and permittivity. It then<br />
extrapolates the real and imaginary parts of permittivity for the specified frequency range<br />
by using a multi-pole Debye relaxation fit that guarantees causality by enforcement of the<br />
Hilbert transform on the real and imaginary parts of permittivity. The electrostatic field<br />
solver outputs a SPICE syntax tabular w-element circuit model that allows the user to<br />
specify the length of the transmission line. A typical cross-section of the stripline<br />
transmission lines is shown in Figure 11.<br />
Very High Density Metric (VHDM) connectors were selected for the system design,<br />
however an exact model of the VHDM connector was unavailable so a similar high<br />
speed connector was modeled and simulated using a full wave 3D electromagnetic field<br />
solver to generate the connector model from DC to 20GHz. New technologies within the<br />
solver, including an improved port solution algorithm for driven terminal modes<br />
combined with a refined adaptive, finite element method mesh solver, enable solutions to<br />
be extracted to DC very accurately. Finally, the vias and SMP connectors were included<br />
in the full-wave, 3D simulations.<br />
Figure 11: Typical stripline cross-section model from the 2D electrostatic field simulator.
Figure 12: Differential via model.<br />
With all of the building block models completed the Ansoft Designer environment was<br />
used to create a system level schematic of the entire BladeServer system including the<br />
PWBs, connectors, vias, drivers, and receivers.. The final pre-layout schematic is shown<br />
in Figure 13.<br />
0 1 2 3 4 5 6 7 0<br />
ref<br />
0<br />
Port1<br />
switch<br />
smp_p HFSS sw_p<br />
1<br />
sw_p HFSS conn_p HFSS<br />
+ +<br />
conn_p HFSS mid_p<br />
1<br />
mid_p HFSS conn_p HFSS<br />
+ +<br />
conn_p HFSS blade_p<br />
1<br />
blade_p HFSS smp_p<br />
blade<br />
Port2<br />
Port3<br />
switch<br />
smp_n<br />
Switch<br />
ref<br />
sw_n<br />
2<br />
sw_n<br />
Switch<br />
ref<br />
conn_n<br />
- -<br />
VHDM<br />
ref<br />
conn_n<br />
mid_n<br />
Mid-Plane<br />
ref<br />
2<br />
mid_n<br />
conn_n<br />
Mid-Plane<br />
ref<br />
- -<br />
VHDM<br />
ref<br />
conn_n<br />
Blade<br />
ref<br />
blade_n<br />
2<br />
blade_n<br />
Blade<br />
ref<br />
smp_n<br />
blade<br />
Port4<br />
ref<br />
0<br />
W1239<br />
0 0 0 0 0<br />
W1238<br />
0 0 0<br />
0<br />
0<br />
W1237<br />
0 0 0<br />
ref<br />
0<br />
0<br />
0 = SMP Connectors (HFSS)<br />
1 = Driver-Side Via Stub (Full Stackup HFSS Model)<br />
2 = Switch PCB Model (W-Elements from Q3D/2D)<br />
3 = Connector Model (HFSS)<br />
4 = Mid-plane PCB Model (W-Elements from Q3D/2D)<br />
5 = Connector Model (HFSS)<br />
6 = Blade PCB Model (W-Elements from Q3D/2D)<br />
7 = Receiver-Side Via Stub (Full Stackup HFSS Model)<br />
Figure 13: Final pre-layout channel schematic.<br />
Eye diagrams and BER contours were obtained using w-element models to represent the<br />
switch, backplane, and blade transmission lines while the connectors and vias used threedimensional<br />
models imported from the full wave 3D EM (electromagnetic) solver. The<br />
initial system level model reflects the pictorial diagram shown in Figure 3. As expected,<br />
the entire system model produced a closed eye without any equalization being used;<br />
initially the SMP connectors were not modeled and idealized connectors were used<br />
between the backplane, switch, and processor blade to obtain a quick solution based upon<br />
the design timeline. The initial pre-layout simulation results are shown in Figure 14. The<br />
initial simulation required 5 taps of FFE at the receiver to obtain an open eye with<br />
ref<br />
0
acceptable BER. The native equalization algorithms embedded in the statistical tools<br />
were used to equalize and open the eye , achieving a BER of 1x10 -14 (Figure 14).<br />
A<br />
C<br />
B<br />
D<br />
Figure 14: A- Initial pre-layout simulation without any equalization using QuickEye analysis. B-<br />
QuickEye analysis using 5 taps of FFE at the receiver. C- Contour BER eye plot from VerifEye<br />
analysis using 5 taps of FFE at the receiver. D- Bathtub BER plot from VerifEye analysis with 5 taps<br />
of FFE at the receiver.<br />
The contour eye BER plot shown in Figure 14 C yields BER as a function of eye<br />
magnitude and eye width. The color gradient shows different BER values with the blue<br />
eye corresponding to a BER of 1x10 -14 and the red corresponding to a BER of 5x10 -2 . A<br />
VerifEye analysis of transmit Gaussian random jitter (RJ) was performed to determine<br />
what operating margin was left in the system. Transmit RJ of 3 ps (Gaussian average<br />
value) was incorporated into VerifEye. Using a six sigma standard to include 99.73% of<br />
the jitter the peak RJ is ~ 9 ps with a delta of ~18 ps.
Figure 15: VerifEye contour BER diagram with 3 ps of RJ introduced into the system with 5 taps of<br />
FFE at the receiver.<br />
VerifEye BER results for the final pre-layout channel (Figure 13) that included the SMP<br />
connector and an additional four 50 mil via stubs are shown in Figure 16. This shows that<br />
additional equalization was required to achieve an acceptable system BER; this required<br />
the inclusion of DFE into the receiver along with FFE. The final design showed 8 taps of<br />
FFE and 3 taps of DFE at the receiver were required to equalize the system channel.<br />
A B<br />
Figure 16: A- VerifEye contour BER plot for Figure 13 yielding BERs up to 1x10<br />
bathtub<br />
-12 . B- VerifEye<br />
BER curve shown up to a BER of 1x10 -12 . Both A and B plots are using 8 taps of FFE and 3<br />
taps of DFE at the receiver.<br />
Post-Layout Analysis<br />
The Allegro CAD design was translated into an Ansoft Neutral File format (.anf) using<br />
Ansoft links. The translated PWB layout was then imported into Ansoft’s SIwave fullwave<br />
package and PWB extractor tool. The package and PWB extractor tool was used to<br />
create a Touchstone S-parameter file that included the exact layout of each lane analyzed.<br />
A differential pair for the 10 Gb/s (5 GHz) lane is shown in Figure 17 for the Blade and<br />
switch cards.
A B<br />
Figure 17: A: SIwave isometric view of the extracted differential pair on the Blade card. B:<br />
SIwave top-down view of the Switch card.<br />
The post-layout model is shown in Figure 18 and consists of the three SIwave models,<br />
SMP connector models, and a high speed connector model. A Touchstone file was<br />
created for each PWB in the system (Blade, Switch, and Backplane); these were used to<br />
model the channel in lieu of w-elements and discreet vias. The full-wave package and<br />
PWB extraction tool performs a 2D FEM solution on all planar objects within the PWB<br />
and combines this with a Method of Moments solution for the signal traces. The final<br />
solution incorporates the fabricated board including the stack-up, plane geometries, trace<br />
geometries, components, vias, via padstacks, and frequency dependent dielectrics for<br />
each layer. Upon completion of the simulation, a Touchstone file was exported to the<br />
statistical tools for rapid,statistical analysis.<br />
0 1 2 3 4 5 0<br />
ref<br />
0<br />
Port1 Port2<br />
SWITCH<br />
HFSS<br />
Mid-Plane<br />
HFSS<br />
Blade<br />
switch<br />
switch<br />
Port3 Port4<br />
ref<br />
ref<br />
ref<br />
0<br />
smp+<br />
smp- conn-<br />
ref<br />
conn+<br />
C<br />
sw+<br />
blade+<br />
sw- blade-<br />
ref<br />
0 0 0 0 0<br />
1 = Switch PCB Model (SIwave)<br />
2 = Connector Model (HFSS)<br />
3 = Mid-plane PCB Model (SIwave)<br />
4 = Connector Model (HFSS)<br />
5 = Blade PCB Model (SIwave)<br />
Figure 18: Post-layout circuit model.<br />
The post-layout simulation results show a very similar solution space to that of the final<br />
pre-layout simulation (2 ps difference at a BER of 1x10 -12 ). VerifEye BER results are<br />
shown in Figure 19 for the post-layout simulation.<br />
C<br />
conn+<br />
smp+<br />
conn- smp-<br />
ref<br />
blade<br />
blade<br />
ref<br />
ref<br />
0<br />
0
Figure 19: Post-layout results for the extracted Switch-Backplane-Blade channel extracted from<br />
SIwave<br />
using VerifEye analysis with 8 taps of FFE and 3 taps of DFE at the receiver.<br />
Measured Analysis<br />
To determine if the simulated solution space was representative of the actual design an<br />
end-to-end frequency domain channel measurement was made and the measured<br />
Touchstone file was used to determine the channel BER using the statistical simulation<br />
tool. The measurement methodology used Gore 3.5 mm (EN0CB0CA036) cables along<br />
with a 3.5 mm to SMP adapter cable (Rosenberger 101076-4604) that allowed mating to<br />
the female SMP connectors on the Switch and Blade. A SOLT calibration was performed<br />
on an Agilent E8361A PNA with an N4421BH67 test switch up to 20 GHz to the end of<br />
the Gore cables and therefore did not calibrate out the 3.5 mm to SMP adapter cable. This<br />
adapter cable has not been taken into consideration in any of the simulations. The<br />
measurement was taken from 10 MHz to 10 GHz with a frequency step of 10 MHz. The<br />
authors realize this measurement methodology will not provide the most accurate data.<br />
However, given the physical size and constraints of the entire system which included:<br />
three PWBs along (Switch, Backplane, and Blade), a right angle connector, and a left<br />
angle connector the measurement setup was considered sufficient. The awkwardness of<br />
the physical setup precluded the use of RF probes with which extremely accurate<br />
measurements could have been made. Due to limitations in the measurement setup the<br />
authors limited the upper frequency of the measurement to 10 GHz and believe error<br />
introduction started to become a factor beyond 6 GHz.<br />
The measured .s4p file was imported into the circuit simulator tool and then simulated<br />
using VerifEye with 8 taps of FFE and 3 taps of DFE at the receiver. Comparing the pre-<br />
and post-layout simulations to the measured channel .s4p simulation good correlation was<br />
obtained. For a BER of 1x10 -12 the measured s-parameters yielded an eye opening of 43<br />
ps where the pre- and post-layout simulations yielded eye opening of 49 ps and 51ps<br />
respectively. The authors believe even better correlation can be obtained by incorporating<br />
the actual VHDM connector model and obtaining better measurement results up to 15<br />
GHz for the simulation. However, despite these anomalies good correlation (8 ps for a<br />
BER of 1x10 -12 ) was achieved. These results have been used in defining an operating<br />
BladeServer KR channel (5 GHz or 10 Gb/s) which includes over 26 inches of FR4 with<br />
very large via stubs (up to 125 mils) within the channel and an additional four 50 mil via<br />
stubs to support the measurement methodology.
Figure 20: BladeServer frequency domain test setup.<br />
Figure 21: VerifEye simulation using the VNA .s4p measurement file.<br />
Conclusion<br />
This paper has demonstrated the ability to accurately predict system level performance<br />
for an extremely harsh 10 Gb/s Ethernet architecture that included 3 PWBs, 2 connectors,<br />
more than 26 inches of standard FR4 (tan δ ~.021), 125 mil via stubs using pre- and postlayout<br />
simulation methodologies in both the time and statistical domains. The results<br />
show very good correlation between measured and simulated data. It has been shown that<br />
pre- and post-layout simulations correlate to within 8 ps of the measured channel at a<br />
BER of 1x10-12. In order to increase the correlation, models of the transitions and<br />
connectors should be included. Inclusion of these models is critical to increase accuracy<br />
of the simulation above the achieved 8%. This will be addressed in future simulations.
The paper describes a BladeCenter along with the advantages it possesses over traditional<br />
rack mount server arrangements. The paper provides an in depth analysis of the elements<br />
required to design and develop 10 Gb/s channels used in such a system.<br />
Finally, the paper introduces two new statistical tools, VerifEye and QuickEye, that<br />
address the industry-wide challenge of impossibly long transient measurement times<br />
implied by current BER requirements and traditional methods. These statistical tools<br />
complement an advanced suite of circuit simulation tools which are used to model the<br />
system performance of a detailed BladeServer channel. An overview of the modeling<br />
algorithms for QuickEye and VerifEye is discussed along with the implementation of<br />
FFE and DFE within each analysis technique.<br />
Acknowledgements<br />
The authors would like to extend thanks to Wahid Ahmed, Michael Tsuk, Matt<br />
Commens, Rich Hall, François Pradeau, Rob Holoboff, Larry Williams, and Bill Kofoed<br />
for their invaluable help in providing support and technical guidance.<br />
References<br />
[1] VHDM Backplane Connector System; see<br />
http://www.molex.com/cmc_upload/0/000/0–8/388/tab01vhdm.pdf<br />
[2] P. Patel, B. Herrman, J. Hughes, J. Wong, and M. Cobo, ‘‘Gigabit Ethernet Allegro<br />
PCB SI Simulation Matching Lab Measurements’’,<br />
http://www.cadence.com/community/allegro/Resources/resources_pcbsi/si/TPIBM_gigab<br />
itsimulation.pdf<br />
[3] Plateau HS Mezz Connector System; see<br />
http://www.molex.com/cgibin/bv/molex/index_login.jsp.<br />
[4] A.R. Djordjevic, R.M. Biljic, V.D. Likar-Smiljanic, T.K. Sarkar, “Wideband<br />
Frequency-Domain Characterization of FR-4 and Time-Domain Causality,” IEEE Trans.<br />
on Electromagnetic Compatibility, V. 43, n4, November 2001, pp 662 – 667.<br />
[5] Anthony Sanders, Mike Resso, John D.Ambrosia, Channel Compliance Testing<br />
Utilizing Novel Statistical Eye Methodology, <strong>DesignCon</strong> 2004.<br />
[6] R.I. Mellitz, M. Tsuk, T. Donisi, S.G. Pytel, “Strategies for Coping with Non-linear<br />
and Time Variant Behavior for High Speed Serial Buffer Modeling”, <strong>DesignCon</strong> 2008,<br />
February 4 – 8.