in PDF Format - Embedded-Control-Europe.com

ATCA, MTCA & AMC
PCI Express interconnects
in high-performance xTCA systems
By Gene Juknevicius and Joerg Hollerith, GE Intelligent Platforms
This article shows that the
PCIe interconnect offers a
number of advantages
compared to other highperformance
computing
interconnect options.
n PCI Express (PCIe) is the native interface in
many processor architectures including most
popular x86 processors, and often is perceived
as a chip-to-chip interface. Other interface options,
such as Ethernet, InfiniBand, and Fibre
Channel, require additional controller devices
that are connected to CPU via PCIe, hence
add extra cost and power and have higher latency
and lower throughput. Consequently, it
makes a lot of sense to use native PCIe wherever
possible. In the past, PCIe interface use was
limited to connecting the processor to I/O devices.
Today, however, PCIe technology allows
a broader spectrum of usage models far beyond
simple I/O connectivity. PCIe is ideally suited
to interconnect multiple processors not only
on the same board, but also on different boards
or even enclosures.
The PCIe interconnect offers a number of
technical advantages, including low latency,
high data throughput, low CPU utilization,
low cost of implementation, and low power
consumption. Other connectivity options,
such as Ethernet and InfiniBand are implemented
using PCIe, meaning that in terms of
the basic communication properties, they will
always be at a disadvantage compared to
native PCIe between boards. Obviously, using
native PCIe does not suit every application.
Requirements such as connectivity over long
distances (more than ~25m), scalability to a
large number of interconnected devices and
application interfaces, will limit the use cases
for PCIe. Today, however, PCIe can do much
more than simply connect the processor to
I/O devices. Non-transparent port operation,
the primary feature that expands usability of
PCIe, can be used to interconnect multiple
processors, each acting as a root complex.
Such connectivity allows a processor memory
window to be mapped into the memory space
of another processor. This interface enables a
basic, but very high-performance communication
model. Unfortunately, at this time there
are no well-accepted APIs for the applications
to use for this communication model, but
that is about to change. A network direct
driver for Windows, currently in development,
will enable the native PCIe connectivity option
to be used for high-performance computing
clusters and other applications.
PCIe has already an important role as an interconnect
option in ATCA and MTCA environments.
In the ATCA world, PCIe is a valid
backplane interconnect option, however, Ethernet
dominates for backplane connectivity.
Certainly, standard PCIe-based ATCA blades
and hubs can be developed but significant engineering
effort is required. On the other hand,
and by customizing ATCA backplanes, removing
unnecessary ATCA features and designing
specific ATCA-like blades, implementers can
June 2011 10
create high-performance cost-effective systems.
Furthermore, most current standard ATCA
single board-computers (SBCs) route one or
more PCIe interfaces to a rear transition module
(RTM) via the Zone 3 connector. ATCA
supports an optional Zone 3 backplane that
can be used to provide an additional interconnect
option between the standard Ethernetbased
ATCA blades.
In the MTCA world, the PCIe interconnect is
very popular. A number of AdvancedMCs
(AMCs) as well as MicroTCA carrier hubs
(MCHs) on the market support PCIe connectivity.
Furthermore, MTCA supports both
point-to-point interconnect between different
boards as well as interconnection via a PCIe
switch on the MCH. The following implementation
example of a control system for the
semiconductor manufacturing equipment highlights
PCIe features and usage models.
Semiconductor manufacturing equipment is
typically size-restricted, and sometimes has
higher environmental requirements for temperature,
shock and vibration. Additional environmental
restrictions also may be imposed
if part of the control system is mounted onto
movable components. For most semiconductor
control systems, communication throughput
and latency in particular are very important.
Such a controller might consist of a few proces-