Fabric Manager Users Guide, Version 6.1, Revision A - QLogic

2–Advanced Fabric Manager Capabilities
Mesh/Torus Topology Support
Path Record Query
Virtual Fabrics
As has been implied by the previous discussion, the dor-updown algorithm uses
multiple LIDs, SLs and VLs. For applications to use the correct route through the
fabric, they must use SA PathRecord queries to obtain the addressing information
for communicating to another Channel Adapter. Unlike simpler algorithms,
“cheating the standard” and bypassing the SM will result in non-optimal
performance. Techniques such as out of band LID exchange (which is used by
many MPI implementations) will provide sub-optimal performance.
To permit non-InfiniBand compliant applications (such as the existing MVAPICH
and OpenMPI implementations for verbs) to function in a Mesh/Torus fabric, the
QLogic Fabric Manager configures the Base LID and the 1st SL on each Channel
Adapter for the Up/Down route. This route will provide reliable deadlock free
operation, even if Channel Adapters simply exchange LIDs. It will also operate
both for complete and disrupted fabrics. However, this route will provide greatly
increased latency and reduced bandwidth as compared to proper use of the
SM/SA.
Many applications use IPoIB for path resolution. Since IPoIB makes PathRecord
queries, such applications will be given optimized routes and will function properly.
To permit optimized MPI performance for Mesh/Torus fabrics, the QLogic Host
Channel Adapter with its Performance Scaled Messaging (PSM) technology
should be used. PSM can perform PathRecord queries when its path_query
option is enabled (see QLogic OFED+ Host Software User Guide).
To ensure scalability when using QLogic PSM with PathRecord queries enabled,
the QLogic Distributed SA (qlogic_sa) must be enabled on every compute node.
The Distributed SA synchronizes the node relevant PathRecord information with
each end node such that job startup time is optimized. See the QLogic OFED+
Host Software User Guide for more information on the Distributed SA.
The QLogic Fabric Manager permits use of Virtual Fabrics in a Mesh/Torus fabric.
In such environments QoS and/or Security can be enabled to separate various
applications or nodes. The QoS Options table in Appendix D shows some of the
combinations of QoS options which are possible. For more information on Virtual
Fabrics refer to Section 4.
Dispersive and Multi-Path Routing
Mesh/Torus fabrics require at least 2 LIDs per Channel Adapter. The LMC is set to
1 by default. One LID is used for the optimized DOR route and one LID is used for
the fall back Up/Down route
Often it is desirable to use advanced features such as dispersive routing in MPIs
using QLogic PSM or other applications which may be able to take advantage of
multiple paths for redundancy, load balancing or performance.
IB0054608-01 B 2-13