Real-time Capabilities for Linux SGI REACT™ Real-Time for Linux®

White PaperReal-time Capabilities for LinuxSGI REACT Real-Time for Linux®

AbstractThis white paper describes the real-time capabilities providedby SGI REACT Real-Time for Linux. software. REACT enablesSGI Linux-based system platforms to deliver hard real-time(HRT) capabilities including an interrupt response time of 30usor less. A brief review of current real-time Linux capabilities, adescription of typical real-time applications, and a comparisonof SGI real-time capabilities for Linux versus IRIX are alsoincluded.II

Table of ContentsAbstract.................................................................................................................... II1.0 Introduction....................................................................................................... 12.0 Real-time Systems And Linux.......................................................................... 13.0 Real-time Applications...................................................................................... 14.0 React For Linux Features................................................................................. 14.1 Altix Real-Time Foundation.............................................................................. 24.2 Hardware-specific Features............................................................................. 2High Resolution Clocks and Timers................................................................ 2External Interrupts............................................................................................ 2VME Support...................................................................................................... 34.3 Unique REACT Software Capabilities............................................................. 3User Level Interrupts........................................................................................ 3Frame Rate Scheduler...................................................................................... 3Kernel Barrier Facility....................................................................................... 3SGI-shield.......................................................................................................... 3SGI Real-time Library........................................................................................ 4Configuration Tools.......................................................................................... 4SGI LINUX Trace (SLT)...................................................................................... 45.0 MIGRATING REAL-TIME APPLICATIONS FROM IRIX TO LINUX................... 56.0 CONCLUSION.................................................................................................... 67.0 REFERENCES.................................................................................................... 6

1.0 IntroductionThe ability to perform computations in real time is critical toa variety of applications. In casual use, people often refer toany computing system capable of returning results quickly as“real-time”, but such systems do not guarantee the latencyof each computation. True real-time processing requires thatcomputations return the correct result within a set period of time.An important distinction in real-time processing is thedifferentiation between hard real-time (HRT) and soft real-time(SRT). In hard real-time systems, the computation is successfulonly when the answer is accurate and it is delivered within therequired time constraint. This time constraint is often referredto as the “guaranteed time.” In soft real-time systems, thecomputation is successful if the answer is accurate and it isdelivered within a threshold of time. HRT systems require specificcapabilities within the operating system that have guaranteedlatencies and determinism; making HRT much more challengingto achieve than SRT.The SGI approach to real-time delivers desirable latencyproperties in a standard Linux user process without requiring aspecial kernel or the compiling of user code inside the kernel.SGI has been producing systems capable of HRT for over tenyears. SGI REACT Real-Time for Linux® is the SGI real-timeextension for SGI Linux platforms. With REACT, SGI platformsdeliver hard real-time interrupt response times—the time thatpasses between the instant a hardware device raises an interruptsignal and the instant the system returns control to a userprocess. REACT for Linux provides significant new capabilitiesthat allow SGI Linux-based systems to sustain interrupt responsetimes of 30µs or less. Such guarantees are important to real-timeprograms because they put an upper bound on the overheadof servicing interrupts and thus make computations morepredictable. REACT is supported on systems ranging from 2to 64 processors in size. Competing HRT systems are typicallylimited to no more than 8 processors. In addition, REACT forLinux running on SGI platforms provides:• Large memory support• Bandwidth and latency advantages of NUMAlink technology• Independent scaling of CPU, memory and I/O• Hardware-dependent external interrupt capability• User-level interrupt handling• Frame Rate Scheduler• High resolution timers/clock• Kernel barrier feature• sgi-shield• SGI Linux Trace• Dynamic reconfiguration capabilityThis paper provides an overview of real-time capabilities onLinux, looks at typical real-time applications, examines the uniquefeatures of SGI REACT Real-Time for Linux, and provides acomparison between the real-time capabilities of SGI REACT forLinux versus IRIX.2.0 Real-Time Systems and LinuxAs is often the case, initial real-time system development beganwith proprietary operating systems. These were often extensionsof other proprietary operating systems used for more generalizedapplications. This was the case with the real-time extensions thatSGI created for its SGI IRIX operating system. More recently, theopen source advantages as well as the “lighter” initial weight ofLinux has favored its adoption and use in real-time systems. Thewidespread use of Linux for Internet-based applications and theneed for embedded systems in Internet and telecommunicationshardware has provided additional impetus.SGI has leveraged its ten plus years of experience with real-timedevelopment on the SGI IRIX operating system in the developmentof SGI REACT for Linux. REACT for Linux combines the basecapabilities of the standard Linux kernel with the unique HRTcapabilities necessary to go beyond the capabilities of previousLinux-based real-time systems to create a complete solutioncapable of tackling the most difficult real-time applications.3.0 Real-Time ApplicationsApplications that require real-time computation are many andvaried. SGI has significant experience with real-time applicationsincluding:• Man-in-the-loop simulators. Closed-loop simulators used foroperator training (aircraft, ship, tank), engineering simulators,and power plant simulators.• Hardware-in-the-loop simulators. Closed-loop simulators withhigh frame rates (thousands of Hz) often requiring high-speed,low-level I/O.• Telemetry, radar, and data acquisition. Open-loop systems foracquiring, processing and storing (or re-transmitting) data inreal-time.• C4I applications (Command, Control, Communications, andIntelligence). Systems that combine a lot of the previousapplication needs and often include displays that need to beupdated for the latest information.• Video applications. Systems that handle live video such asvirtual sets, real-time broadcast, and video-on-demand (VOD)4.0 REACT for LINUX FeaturesThe SGI Linux-based platform runs on standard Linuxdistributions including SUSE® Linux Enterprise. The REACTsoftware consists of modules that run on top of the standardSUSE Linux kernel (but do not replace the kernel), providing avariety of additional features that enhance real-time execution.1

Figure 1.Componentsof interruptresponse.4.1 Altix Real-Time FoundationThe key characteristic of REACT for Linux is its ability toguarantee an interrupt response time of 30µs on systems rangingfrom 2-64 processors. The total interrupt response time includes:• Hardware latency. The time required for a CPU respond to aninterrupt signal.• Software latency. The time required to dispatch an interruptthread.• Device service time. The time the device driver spendsprocessing the interrupt and dispatching a user thread.• Mode switch. The time it takes to exit kernel mode andresume execution of the user thread.REACT ensures that the combined time for all these activities willnot exceed 30µs on a properly configured system.A variety of factors contribute to this response time including:• At system startup, REACT performs a number of configurationoperations such that each CPU designated as a realtimeCPU provides minimal thread latency. This includesappropriate redirection of all I/O interrupts to non real-time CPUs.• REACT for Linux includes features that provide programmaticand command-line access to features that allow processpinning, dynamic real-time CPU configuration, schedulerprioritization, memory locking, and dynamic I/O interruptredirection.• I/O interrupts to be serviced by real-time threads can beredirected to real-time CPUs. In general, real-time CPUsshould only service interrupts from devices critical to theexecution of the associated application(s). The interruptservice routine (ISR) that handles the interrupt should do verylittle processing and should avoid time-consuming operationssuch as allocating buffers.• By default, CPUs are not isolated from system timerinterrupts since the system timer typically produces no majorlatency issues. (CPU 0 always performs system-wide clockprocessing in ia64 Linux.) If desired, sgi-shield can be usedto isolate CPUs from system timer interrupts. This eliminatesjitter and gives the user application complete control of theCPU such that a CPU can be dedicated to run only thedesired user code.A number of additional REACT features ensure correct andpredictable real-time operation and facilitate the creation andexecution of real-time applications.4.2 Hardware-specific FeaturesREACT for Linux includes the ability to take advantage of anumber of capabilities unique to SGI hardware.High Resolution Clocks and TimersSGI Altix provides a system-wide clock called a real-time clock(RTC) that is accessible to every CPU. The RTC provides a rawtime source that is incremented in 50-ns intervals. High resolutionPOSIX timers on Altix utilize the RTC as a time source.The RTC is 55 bits wide, which ensures that it will return a uniquevalue for more than half a century. RTC values are mappedinto the local memory of each processing node. Multiple nodesaccessing the RTC value do not reduce the performance of theclock functions. The RTC is synchronized among all of the nodesin an SGI system using a special pin on the NUMAlink cableproviding superior scalability. This feature is not available on Altix XE.External InterruptsReal-time processes often require the ability to respond toexternal events. The external interrupt capability of SGI Linuxbasedsystems allows a real-time process to receive an externalsignal. An external interrupt is generated when an electricalsignal is applied to the external interrupt socket on the SGIPCI-RT-Z card. This card has an external jack into which a 0-5V signal can be fed. External equipment can assert this line,2

causing the card to generate an interrupt. The I/O card also hasthe ability to generate interrupts for signaling external equipmentusing the same external jack. This capability enables SGI Linuxsystems to coordinate activities with external devices or other,independent, SGI systems.VME SupportMany real-time system designs require the ability to interface toexternal VME equipment. VME solutions are available from thirdparty vendors through SGI Professional Services.4.3 Unique REACT Software CapabilitiesSGI systems running REACT are able to utilize all of the real-timecapabilities of the standard Linux distribution. SGI continues todevelop and contribute to the real-time capabilities of Linux. Inaddition, REACT provides some unique capabilities.User Level InterruptsThe user-level interrupt (ULI) facility allows a hardware interruptto be handled by a user process. A user process may registera function with the kernel, linked into the process in the normalfashion, to be called when a particular interrupt is received. Theinterrupt handler is called asynchronously and has access only tothe process’s address space. A significant advantage of the ULIfacility is that it can be very fast. Applications that need to handlespecific interrupts in less than the 30 microsecond maximumguaranteed by REACT utilize this feature. The best case is aresponse in 3-4µs. The worst case is no slower than the system’sworst case for scheduling a process.User LevelUserProcessInterrupt LevelULIfunctionKernalISRInterruptFigure 2. User-level interrupt (ULI) handling.The kernel ISRcalls the functionresiding in userprocess spaceThe ULI facility is also intended to simplify the creation of devicedrivers for unsupported devices. ULIs can be written to respondto interrupts initiated from external interrupt ports. A programmingerror in the driver will result in nothing more serious than thetermination of a process rather than crashing the entire system,and the developer need not know anything about interfacing adriver into the kernel.The ULI feature may also be used for high-performance I/Oapplications when combined with memory-mapped device I/O.Applications can make all device accesses in user space. This isuseful for high-performance I/O applications such as hardwarein-the-loopsimulators.Frame Rate SchedulerThe Frame Rate Scheduler makes it easy to structure a real-timeprogram as a family of independent, cooperating activities thatare running on multiple CPUs and are scheduled in sequence atthe frame rate of the application. The frame rate scheduler cansynchronize the activities of many CPUs, while providing flexiblenotification and schedule changes during over and under runs.Instead of scheduling threads according to priorities, the framescheduler dispatches them according to a strict, cyclic rotationgoverned by a repetitive time base. The time base determinesthe fundamental frame rate. Some examples of the time baseinclude:• A specific clocked interval in microseconds• An external interrupt• A device interrupt from a specially modified device driverThe interrupts from the time base define minor frames. Together,a fixed number of minor frames make up a major frame. Thelength of a major frame defines the application’s true framerate. The minor frames allow you to divide a major frame intosubframes.The frame scheduler maintains a queue of threads for each minorframe. Each activity thread of the program is queued to a specificminor frame. The order of cyclic execution within a minor frame isdetermined by the order in which threads are queued.Kernel Barrier FacilityThe kbar kernel barrier facility provides for the fast wake-up ofmany blocked user threads. When the barrier is signaled, theoperating system will use a configurable number of CPUs toquickly wake all blocked threads. A maximum of 64 barriers aresupported system-wide. Kbar barriers do not behave preciselythe same as traditional barriers. A specific number of threads donot have to be blocked on the barrier for them to be woken.SGI-shieldThe sgi-shield feature allows CPUs to be isolated from systemtimer interrupts. This eliminates jitter (short interruptions inprocess execution) which gives a user application completecontrol of the CPU. Shielding can be turned on dynamically fromwithin the application and should only be used for short periodswhere jitter-free program execution is required.3

Figure 3.Relationshipbetween majorand minorframes andthread queues.SGI Real-time LibraryREACT for Linux includes a userspace library called libsgirtthat contains IRIX compatibility routines and other user spaceroutines unique to REACT. This library provides a set ofinterfaces similar to those found in previous-generation SGIIRIX real-time offerings to ease the migration path for customersmoving real-time applications from IRIX to Linux. Using the librarycan also insulate users from some future changes in the real-timeoperability and configuration of Altix systems.Configuration ToolsREACT provides a script that is used to configure systems forreal-time processing. This script performs CPU isolation andallows dynamic reconfiguration of real-time CPU assigments.SGI LINUX Trace (SLT)SLT is a debug kernel that can generate traces of kernelevents such as interrupt handling, scheduling and systemcalls. Recorded trace events are viewable with the graphicaltracevisualizer tool, allowing users to analyze how kernelbehavior impacts the execution of real-time applications.5.0 Migrating Real-Time Applications from IRIX to LinuxSGI has made every effort to provide comparable capabilitiesbetween its real-time extensions for IRIX and Linux to simplifymigration. Both real-time implementations provide such features as:• User level interrupts• Frame rate scheduler (API is nearly identical)• Debug tools• Dynamic reconfiguration• External interrupts• Programmatic and command line access to features that allowmemory locking, process pinning, and CPU restriction andisolationIn addition, REACT for Linux adds unique capabilities includingsgi-shield and kbar that were not available on IRIX. The Linuxbasedplatform delivers better real-time performance than SGIOrigin running IRIX with realtime extensions: 30µs guaranteedinterrupt response time versus 50µs for Origin.Figure 4.SLT output.4

6.0 ConclusionSGI has been an industry leader in the real-time arena for overa decade. SGI is actively participating in the Linux community,identifying hold-offs and submitting modifications that result inreduced latencies. SGI delivers expertise and support and setupinformation that can’t be found anywhere else.Our long term strategy is to continue to work with the Linuxcommunity to enhance, fix and test the real-time capabilities ofthe standard Linux kernel. SGI tests the Linux kernel on a weeklybasis to ensure that no regressions end up in the distribution. Inaddition, SGI will continue to add specific capabilities to REACTfor Linux to address specific customer requirements.7.0 References1. http://www.sgi.com/products/software/react/linux_overview.html2. http://www.sgi.com/products/software/react/3. http://www.sgi.com/support/custeducation/courses/sw_dev/real-time_programming.html4. SGI® REACT Real-Time for Linux® Programmer’s Guide,(document number 007-4746-004) available at http://techpubs.sgi.com6Corporate OfficeSGI1140 East Arques AvenueSunnyvale, CA 94085-4602650.960.1980North America +1 800.800.7441Latin America +55 11.5185.2860Europe +44 118.912.7500Japan +81 3.5488.1811Asia Pacific +1 650.933.3000© 2006 SGI. All rights reserved. SGI, the SGI cube and the SGI logo are registered trademarks of SGI in the United States and/or other countries worldwide, including but not limited to Altix, NUMAlink, andSGI MIPS processors. All other trademarks mentioned herein are the property of their respective owners.3993 [2.2007] J15254[11.14.2003] J14415 3925 [3.2006] J15120