Qualification and Reliability of Complex Electronic Rotorcraft Systems

Qualification and Reliability of Complex Electronic Rotorcraft SystemsAlex K. Boydston, MSEE, U.S. Army Electronics EngineerWilliam D. Lewis, Ph.D. Director of U.S. Army Aviation Engineering DirectorateRedstone Arsenal, AL,alex.boydston@us.army.milbill.lewis6@us.army.milExecutive SummaryThere is a need to develop an industry standardmethod for qualifying complex integrated systems toa specified reliability. The goal of this paper is not toresolve the problems or evaluate past or existingdesigns, but it is to catch the attention of governmentand commercial aviation industry and solicit inputinto new processes and standards to correct thelongstanding issues in the development of complexavionics systems. It will encourage committeeformation to address these issues. The United States(U.S.) Army Aviation Engineering Directorate(AED) requests the input of all technical stakeholdersand desires to not dictate these requirements but toestablish requirements, tools and guidelines forreliability and qualification as a unified community.AbstractMilitary, civil and commercial rotary wing and fixedwind aircraft rely on complex and highly integratedhardware and software systems for safe operation andsuccessful execution of missions. These complexsystems are now classified as “cyber-physicalsystems” per the National Science Foundation (NSF)[38]. Whatever the architecture chosen, be itfederated, integrated or some hybrid, complexavionics systems must be robust, reliable and safe.The architecture should meet the functionalrequirements for timeliness, predictability andcontrollability in order to satisfy the needs to safelyand effectively aviate, navigate, communicate, andexecute a mission.Traditionally, avionics systems were federated, buthave evolved into highly integrated systems ofcomputer hardware and software. In an integratedsystem, if the architecture has flaws, faults cancouple between systems and propagate, leading tounpredictable and unreliable behavior unless properpartitioning is accomplished. Qualification andreliability assessment of such systems is challengingwithin schedule and budget constraints despite usingaccepted engineering practices. Given this, there is aneed to develop an industry standard method forqualifying complex integrated systems to a specifiedreliability.Avionics systems require deterministic performance forcritical functions. Integrated Modular Avionic (IMA)systems introduce new issues with data processing, errorchecking, and error handling beyond their federatedcounterparts. Federated systems were not tightly coupledand seemed easier to test to some degree since thefunctions were encapsulated in dedicated and separateunits. Now, avionics systems rely on computer hardwarerunning multiple processes on an integrated system whichhopefully is implemented with a partitioned operatingsystem. Whether federated, integrated or some hybridcyber-physical system, it is crucial for reliability, safety,verification and validation to be embedded in the life-cycleof a system. Dealing with this complexity is challengingbecause of the intricacies present in the design of suchsystems. Stand-alone federated systems have reliabilitytargets, based upon their functions. Complex cyberphysicalsystems may have both distributed and integratedfunctions and multithreaded processes.Program management, systems engineers, developers,systems integration engineers, human factors engineers,test engineers and other disciplines must be cognizant ofthe need for design for testability and high reliability ofsuch systems early in the life-cycle. Waiting untilPreliminary Design Review (PDR) is too late to startaddressing these considerations and may require redesignlater or onerous testing during qualification that is costlyand time consuming. If complete and correct requirementsare not given emphasis at the high level then problemswith poor reliability and inadequate qualification are foundafter the systems are acquired by the user. This may resultin low performance and, possibly, risk of injury or loss oflife.Processes for project management and engineering arefairly well understood and promoted through the defenseacquisition guidelines and by various industry standardssuch as the Capability Maturity Model Index (CMMI),MIL-STD-882, SAE ARP4754, SAE ARP4761, SAEARP4754, DO-178B, DO-254, ARINC 653 and others.Taken together, these standards provide guidance that, iffollowed, likely will result in safe, highly reliable and costeffectivesystems over the life-cycle of the system.Typically, as complexity and redundancy of systemsincrease, the reliability also increases to a break-even pointand then begins to decrease at some level of complexity.2

Redundant systems by their nature increasecomplexity of the system. They are designed withthe goal to provide fault tolerance, which ultimatelyprotects the crew, passengers, test personnel, groundpersonnel, bystanders, and expensive equipment.Correct decisions must be made on good data withproper timing, data type and data exchange in mind.As the complexity and integration of the systemincreases, costs in design and qualification andschedule sometimes increases.The current guidelines indicate how bad a design is,but do not define how to measure the expectedreliability of such complex systems. Additionally,these guidelines fall short in defining how reliabilityis measured throughout the life-cycle of a program.The current system design guidance often does notyield desired reliability of complex electronic aircraftgoals of meeting performance specifications. Thismust be introduced at higher levels like the ProgramElement Office (PEO) because they control overallcost and schedule. Presently, ease of test,reliability and safety are by-products of thedesigners‟ and implementers‟ diligence in meetingpurposeful requirements. There must be a concertedeffort from government agencies such as the FederalAviation Administration (FAA), Department ofDefense (DOD), and commercial aviation industry todefine better design guidance. These proposedguidelines should not constrain the creativity indesign, but must set the boundaries necessary forachieving advances in design with performance,reliability and safety in mind.In summary, this paper will:1) Identify development challenges with complexsystems,2) Present the current U.S. Army approach toairworthiness and testing,3) Discuss the challenges facing reliability andsafety,4) Survey complexity issues,5) Cover the historical aspect of the reliabilityproblem and identify that this not a new problem,6) Itemize some current design guidelines formodeling systems and identify deficiencies,7) Address certification assessment considerations,8) Encourage development and refinement ofstandard modeling and analysis tools to place powerin the hands of the designer to mitigate issues withsystem reliability early and throughout the projectlife-cycle, and9) Request input from the development and testcommunity to establish standard processes andrequirements for qualifying complex systems.Development Challenges with Complex SystemsIt would be wonderful if all systems were straightforwardin design, easily testable and simple to write anAirworthiness Release (AWR) for; however, that is not thecase. Legacy aircraft have been upgraded in a piecemealfashion, acquiring much needed improvements in aviation,navigation and communication. The generalrecommended system development V-curve as shown inFigure 1 is not always followed in a strict sense; although,it should be the goal process.Figure 1 - Defense Acquisition System Design and TestCycleNegligence to the proper process makes the establishmentof certification very difficult. For new systemdevelopment and existing system upgrades, requirementsmust be clear, complete and testable. The certificationrequirements must be made obvious in the development ofthe requirements establishment phase; with the goal ofbeing fully identified during the requirementsdevelopment. Orchestrating agreement among allstakeholders (e.g., the program manager, systemsengineers, human factors engineers, integrators, testengineers, manufacturers, users, and certifiers) is necessaryto mitigate problems such as:juggling multiple software builds,producing a difficult-to-test, difficult-to-certify, anddifficult-to-deploy systems,misunderstanding system safety, andrequiring design iterations that impact schedulesand costs.Specifications are sometimes loaded with so-called “bloat”or “feature creep” that is not necessary for achieving themission. As mentioned by Lui Sha, “useful-butunessentialfeatures cause most of the complexity. [The]approach should combine complexity control architecturedesigns with formal methods, so that the architecturedesign ensures the simplicity of: (1) the critical service,3

modifications, to function satisfactorily when usedand maintained within prescribed limits [25]. TheU.S. Army has traditionally qualified systems andcomponents by similarity, analysis, test,demonstration, or examination. Historically, mostsystems were federated. They were hardware based,simple and distributed. Now they have become moreintegrated. They are more software intensive,complex and have combined functionality containedin one or more computers. With this evolution fromsimple to more complex the Army is finding it moredifficult to execute an AWR. As systems evolve tomore complex systems of systems this problem isonly growing worse.The current test approach to achieving confidence insystems for an AWR for the U.S. Army is basedmore on traditional federated avionics systems.Experienced personnel in Software VehicleManagement Systems, Avionic Communications,Navigation & Guidance Control, Electrical,Environmental, Human Factors, Electrical andElectro-magnetic Effects (E 3 ), Structures, Thermal,Safety, Integration Testing, and Flight Test personneland test pilots all play important roles inaccomplishing test and review of new and existingsystems. While some may not consider areas such asthermal or E 3 important to software development theyare crucial since the software is running on physicalsystems that are affected by heat and susceptibility toelectromagnetic radiation which can cause abnormaloperation or bit errors. Current test methodology forhardware relies on MIL-STD-810, MIL-STD-461,and requirements analysis such as traceability. MIL-STD-810 is the "Department of Defense Test MethodStandard for Environmental EngineeringConsiderations and Laboratory Tests".Figure 4. Software Engineering Directorate (SED)Avionics Software Integration Facility (ASIF)Figure 5 – Kiowa, Apache, Chinook and Blackhawk inTest Flights5

Table 1 – PEO Aviation System Safety Management Decision Authority Matrix(P = Probability based on 100,000 flight hours. Note: Dollar impact values were once associated with this.)Table 2 - Hardware Reliability versus Software Reliability (reference [39])Hardware ReliabilityFailure rate has a bathtub curve. The burn-in state issimilar to the software debugging state.Material deterioration can cause failures even thoughthe system is not used.Failure data are fitted to some distributions. Theselection of the underlying distribution is based onthe analysis of failure data and experiences.Emphasis is placed on analyzing failure data.Failures are caused by material deterioration, designerrors, misuse, and environment.Can be improved by better design, better material,applying redundancy and accelerated life cycletesting.Hardware repairs restore the original condition.Hardware failures are usually preceded by warnings.Hardware components can be standardized.Hardware can usually be tested exhaustively.Software ReliabilityWithout considering program evolution, failure rate isstatistically non-increasing.Failures never occur if the software is not used.Most models are analytically derived fromassumptions. Emphasis is on developing the model,the interpretation of the model assumptions, and thephysical meaning of the parameters.Failures are caused by incorrect logic, incorrectstatements, or incorrect input data.Can be improved by increasing testing effort andcorrecting discovered faults. Reliability tends tochange continuously during testing due to the additionof problems in new code or the removal of problems bydebugging errors.Software repairs establish a new piece of software.Software failures are rarely preceded by warnings.Software components have rarely been standardized.Software essentially requires infinite testing forcompleteness.6

on the design which costs time and money.Furthermore, dealing with existing, fielded complexsystems that have not gone through the rigors of propersystems engineering results, quite often, with anoverwhelming mess of functionality to test, verify,validate, qualify, and certify designs at PDR is too late.It has been shown that discovering issues at this stage inthe life-cycle will cause a reiteration on the designwhich costs time and money. Furthermore, dealing withexisting, fielded complex systems that have not gonethrough the rigors of proper systems engineering results,quite often, with a complicated conglomeration offunctionality to test, verify, validate, qualify, certify andmaintain. This is indicative where complexity hasexceeded our understanding in how to certify a system.We still do not fully understand complexity and how toaddress reliability of complex systems. How do weaccurately assess operating risks, performance, andreliability of complex systems based on limited testingand analysis? How do we know when a system designis good enough? How do we modularize spirallydeveloped systems to minimize the need for requalificationof unchanging portions of the system? Weare 30 plus years into this technology and we still dealwith systems with latent defects that are occurring insupposedly well-tested, and mature systems. To furtherexacerbate the problem we are now dealing withcomplex system of systems (i.e., cyber-physicalsystems).It is a given that you can continue to add redundancyand complexity to a problem at some point in time thereliability will taper off. At best, we sometimes mustsatisfy for an optimum point before digressing inreliability. In the same vein, system development costsand schedule increase with complexity too (see Figure 7and 8).Avionics parts and software constantly change over thelife of a program. Typically a spiral developmentprogram occurs with complex software developmentwhich means that qualification is required frequently.This begs the question of how to streamline the processso that the need to conduct a complete requalification isavoided.With these complex systems there are other hurdles tocross such as fully characterizing and conducting theFunctional Hazard Assessments (FHAs), Fault TreeAnalyses (FTAs) and Failure Mode, Effects, andCriticality Analyses (FMECAs). It is crucial for thesafety assessment that these are conducted correctly tofully understand the risks and later perform the correcttests at the right level. Additionally, once the complexcomponent hardware and software are integrated thenyet other problems appear. It is at that time that thedisparity of teams crossing multiple contractors anddevelopment groups becomes obvious and, if notproperly coordinated, could cause impacts to theschedule. Other programmatic problems affect complexsystem development and qualification. For instance,lack of schedule and funding resources causes ashortcoming to adequately provide for the propercompliance with specification and requirements shortcircuitingthe systems engineering process. An everdecreasing availability of trained engineers to supportthe development and test of such systems exists. Nontechnicalpolitical influences sometimes affect thereliability of complex systems and are difficult to avoid.Lastly, there is a lack of a centralized database thatcaptures the various families of systems that have beenbuilt along with their characterization of success andfailures. Such a database from all past and presentgovernment complex systems could be valuable inestablishing reliability basis for future models.Figure 7 – Notional Trends with Complexity vs.ReliabilityFigure 8 – Notional Trends with Complexity vs. Costand Schedule8

An Old ProblemAs mentioned earlier, complex avionics systems are nota new idea. Since the early 1960‟s complex avionicarchitectures have existed beginning with theApollo/Saturn program. Massachusetts Institute ofTechnology (MIT) Instrumentation Lab (IL), which isnow Draper Laboratory, and International BusinessMachines (IBM) led the way with the MIT/IL ApolloGuidance Computer (AGC) and the Saturn V IBM triplemodular redundant (TMR) voting guidance computersystem. The word software was not even coined at thetime but engineers such as Margaret Hamilton, MIT/ILDirector of Apollo On-board Software, can attest to thefact that some the same issues with creating reliablesoftware then still exists today [5]. A large majority ofthe issues then dealt with the communication betweensystems engineers and the programmers. Requirementswere thrown over the wall without the confirmation thatthe requirements were complete and a lot of the issuescropped up as interface problems. In fact, according toHamilton somewhere on the order of 75% of theproblems experienced in Apollo were interfaceproblems. Identifying these issues prompted Hamiltonto create her own company and create a modelinglanguage called Universal Systems Language (USL) tohead off the problems experienced with Apollo [10, 11].Some 200 plus modeling programs have been developedsince Apollo and used to mitigate issues and increaseconfidence in systems of varying complexity.As time progressed other systems came along. TheNASA Dryden F8 Crusader was the first digital fly bywire (DFBW) jet aircraft that relied heavily on complexIMA and software for flight control. The SpaceTransportation System (STS) shuttle includes a QuadModular Redundant (QMR) system with a fifth backupflight computer containing uncommon code. U.S. AirForce and Naval airplanes that have possessed complexor redundant IMA configurations include the F14Tomcat, F15 Eagle, F16 Falcon, F18 Hornet, F22Raptor, F35 Joint Strike Fighter, F117 Nighthawk, V22Osprey, C17 Globemaster and many more along withrecent Unmanned Air Vehicle Systems (UAVS). TheU.S. Army complex systems on helicopters include the:RAH-66 Comanche DFBW Triple ModularRedundant (TMR) architecture,Glass cockpit avionics on the UH-60M Blackhawkbaseline,Common Avionics Architecture System (CAAS)glass cockpit on the UH-60M Blackhawkmodernization and CH-47F Chinooks and otheraircraft.Full Authority Digital Engine Control (FADEC)units found on various helicopters.Additionally, there are many self-checking pair enginecontroller systems along with system of system FutureCombat Systems (FCS) and Unmanned Air VehicleSystems (UAVS). This has also permeated thecommercial airliner market with the Airbus 320, higherAirbus models, Boeing 777 and Boeing 787 aircraft.With this ever increasing technology something must bedone about the reliability issue. With such a wealth ofdata on aviation and non-aviation cyber-physicalsystems such as submarine, ship, nuclear, medical,locomotive and automotive systems there should beadequate information to get a start on modeling systemscorrectly for reliability. Therefore, this is not anisolated problem to avionics and other disciplinesshould aide in resolving this problem.Along with these examples there have also been manyinstances of failures of complex systems or software. Afew examples include the following:Multiple crashes occurred with the V-22 Osprey[41].In 1999 the Mars Climate Orbiter crashed becauseof incorrect units in a program caused by poorsystems engineering practices [42, 44].In 1988 an Airbus 320 was shot down by the USSVincennes because of cryptic and misleadingoutput displayed by the tracking software [3].In 1989 an Airbus A320 crashed at an air show dueto altitude indication and software handling [3].In 1994 a China Airlines Airbus Industries A300crash on killing 264 from faulty software [3].In 1996 the first Ariane 5 satellite launcherdestruction mishap was caused by a faulty softwaredesign error with a few lines of ADA codecontaining unprotected variables. Horizontalvelocity of the Ariane 5 exceeded the Ariane 4resulting in the guidance system veering the rocketoff course. Insufficient testing did not catch thiserror which was a carry-over from Ariane 4.[3, 39]In 1986 a Mexicana Airlines Boeing 727 airlinercrashed into a mountain due to the software notcorrectly determining the mountain position [39].In 1986 the Therac-25 radiation therapy machinesoverdosed cancer patients due to flaw in thecomputer program controlling the highly automateddevices [3, 39, 45].During the maiden launch in 1981 of the Columbiaspace shuttle, a failure of the primary flight controlcomputer system to establish sync with the backupduring prelaunch [43].In 1997 The Mars Pathfinder software resetproblem due to latent task execution caused bypriority inversion with a mutex [3, 44].An error in a single FORTRAN statement resultedin the loss of the first American probe to Venus.9

From G. J. Myers, Software Reliability: Principles& Practice, p. 25 [3]. In September 1999 the Korean Airlines KAL 901accident in Guam killed 225 out of 254 aboard. Aworldwide bug was discovered in barometricaltimetry in Ground Proximity Warning System(GPWS). From ACM SIGSOFT SoftwareEngineering Notes, vol. 23, no. 1 [3]. The Soviet Phobos I Mars probe was lost, due to afaulty software update, at a cost of 300 millionrubles. Its disorientation broke the radio link andthe solar batteries discharged before reacquisition.From Aviation Week, 13 Feb 1989 [3]. An F-18 fighter plane crash due to a missingexception condition. From ACM SIGSOFTSoftware Engineering Notes, vol. 6, no. 2 [3]. An F-14 fighter plane was lost to uncontrollablespin, traced to tactical software. From ACMSIGSOFT Software Engineering, vol.9, no.5 [3]. In 1989, Swedish Gripen prototype crashed due tosoftware in their digital fly-by-wire system [3, 46]. In 1995, another Gripen fighter plane crashedduring air-show caused by a software issue [3, 46]. On February 11 2007, twelve F/A-22 Raptors wereforced to head back to the Hawaii when a softwarebug caused a computer crash as they were crossingInternational Date Line.[47] February 2006 German-Spanish UnmannedCombat Air Vehicle, Barracuda, crash due tosoftware failure [4]. December 2004 a glitch in the software forcontrolling flight probably caused an F/A-22Raptor stealth fighter jet to crash on takeoff atNellis Air Force [4]. In 2008 a United Airbus A320, registrationN462UA, experienced multiple avionics andelectrical failures, including loss of allcommunications, shortly after departing NewarkLiberty International Airport in Newark, NewJersey [NTSB Report ID: DCA08IA033]. In 2006 a Boeing 777 Malaysian Airlines jetliner‟sautopilot caused a stall to occur by climbing 3000feet. Pilots struggled to nose down the plane butplunged into a steep dive. After pulling back up thepilots regained control. Cause was defective flightsoftware providing incorrect data for airspeed andacceleration, confusing the flight computers, andinitially ignoring the pilot‟s commands.[49] US Army and Air Force UAV crashes from controlsystem or human error.In Dr. Nancy Leveson‟s paper, “The Role of Softwarein Spacecraft Accidents”, she cited problems withsoftware development issues within NASA on variousprojects [37]. According to Dr. Leveson, there were“flaws in the safety culture, diffusion of responsibilityand authority, limited communication channels and poorinformation flow, inadequate system and softwareengineering, poor or missing specifications, unnecessarycomplexity and software functionality, software reuse orchanges without appropriate safety analysis, violation ofbasic safety engineering practices, inadequate systemsafety engineering, flaws in test and simulationenvironments, and inadequate human factors design forsoftware”. While these problems were identified forspacecraft development within NASA and corrected,aviation in general could learn from these lessons tomitigate issues with complex systems development.Current Guidelines and Certification AssessmentConsiderationsThe problems identified establish a need for standardsand guidelines. The aviation community has developedguidelines such as the following to provide a path tocreating robust and reliable aircraft:DO-178B-Software Considerations in AirborneSystems and Equipment CertificationDO-248B–Final Report for the Clarification of DO-178BDO-278-Guidelines for Communications,Navigation, Surveillance, and Air TrafficManagement (CNS/ATM) Systems SoftwareIntegrity AssuranceDO-254-Design Assurance Guidance for AirborneElectronic HardwareDO-297–Integrated Modular Avionics (IMA)Development Guidance and CertificationConsiderationsSAE-ARP4754–Certification Consideration forHighly Integrated or Complex Aircraft SystemsSAE-ARP4671- Guidelines and Methods forConducting the Safety Assessment Process onAirborne Systems and EquipmentFAA Advisory Circular AC27-1B-Certification ofNormal Category RotorcraftFAA Advisory Circular AC29-2C-Certification ofTransport Category RotorcraftISO/IEC 12207-Software Life-cycle ProcessesARINC 653-Specification Standard for Time andSystem PartitionMIL-STD-882D-DoD System SafetyADS-51-HDBK-Rotorcraft and AircraftQualification HandbookAR-70-62-Airworthiness Release StandardADS-75-SS–Army Aviation System SafetyAssessments and AnalysesADS-48-PRF-Performance Specification forAirworthiness Qualification Requirements forOperation of Aircraft in Instrument MeteorologicalConditions and Civil Instrument Flight Rules10

ADS-64-SP–Airworthiness Requirements forMilitary RotorcraftSED-SES-PMHFSA001-Software EngineeringDirectorate (SED) Software EngineeringEvaluation System (SEES) Program ManagerHandbook for Flight Software AirworthinessSED-SES-PMHSS001 SED SEES ProgramManager Handbook for Software SafetyThe current guidelines such as DO-178B, DO-254, DO-297, SAE-ARP-4754, and SAE-ARP-4671 along withmany other guidelines outline the proper steps thatshould be taken. System safety management‟s militarystandard is MIL-STD-882 and has been in use fordecades. Civilian safety standards for the aviationindustry include SAE ARP4754 which shows theincorporation of system safety activities into the designprocess and provides guidance on techniques to use toensure a safe design. SAE ARP4761 containssignificant guidance on how to perform the systemsafety activities spoken about in SAE ARP4754. DO-178B outlines development assurance guidance foraviation software based on the failure conditioncategorization of the functionality provided by thesoftware. DO-254 embodies similar guidance foraviation hardware. ARINC 653 is a widely acceptedstandard to ensure time and space partitioning forsoftware. DO-297 does an excellent job of describingthe certification tasks to take for an IMA system whichinclude:Task 1: Module acceptanceTask 2: Application software/hardware acceptanceTask 3: IMA system acceptanceTask 4: Aircraft integration of IMA systemsincluding verification and validationTask 5: Change of modules or applicationsTask 6: Reuse of modules or applicationsTaken together, these standards provide guidance that, iffollowed, likely will result in safe, highly reliable andcost-effective systems over the life-cycle of the system.Do we have enough standards now, but just do not payto contract them and test to them? Perhaps it should beinvestigated that these standards are consolidated andtailored to a single standard for complex systems similarto how MIL-STD-586 was a consolidation of severalstandards with only applicable parts taken.While these guidelines exist, there is not a consistentindustry-wide method to assess a system at any stage ofthe life-cycle to allow a tradeoff of design alternatives.Also, there is not a standard outlining overall reliabilityfor a system to include hardware and softwarereliability. In order to achieve this level of reliability astandard should be developed to define the process andmethod to achieve a quantifiable reliability number thatwould, in turn lead to acceptance.In order to execute an AWR sufficient data must beprovided to the appropriate airworthiness authorities todemonstrate that safety requirements are met to thelevels per the System Safety Assessment (SSA), FHA,FTA, FMECA or equivalent. The certification processis currently lengthy and depends on much humaninterpretation of the myriad of complex architecturefunctions. Something needs to be done to remedy thiswithout adversely affecting cost and to the programsdeveloping complex systems.Architectural Modeling and Tools to Mitigate IssuesAs mentioned in the introduction the overall goal of thispaper is to state the need for an industry standardmethod for qualifying complex integrated systems to aspecified reliability. While not suggesting as solution intotal, better modeling practices should be consideredtoward a solution to bridge the gap between design, testand implementation. A method to model thearchitecture early in the requirements establishmentphase, follow the detailed design and coding, and thenbe able to verify the system along with the model maybe a path to greater confidence in the system and reducethe risks, warnings and cautions that must be issued.In order to achieve this, the systems must be brokendown to component levels and built up to subsystemand system levels. An overall aggregated systemreliability value should result (see Figure 9). The goalshould be to establish the ability to assess the reliabilityfrom a component, subsystem and then a system levelwith each phase working toward a higher TechnicalReadiness Level (TRL). The end result would be fedinto the accepted Type Certificate (TC) or AWR.To achieve this goal modeling and analysis tools thatfollow a standard for modeling reliability should exist.As previously stated, over 200 tools have been createdsince 1970s. Here is a list of a few of the current tools:Universal Systems Language (USL)Unified Modeling Language (UML)Systems Modeling Language (SysML)MATLAB/SimulinkTelelogic RhapsodyMathCADColored Petri NetsRate Monotonic Analysis (RMA)STATEMATE (Used by Airbus)Standard for the Development of Safety-CriticalEmbedded Software (SCADE)OPNETEmbedded System Modeling Language (ESML)11

Component Synthesis using Model-IntegratedComputing (CoSMIC)Architectural Analysis and Design Language(AADL)By no means is this list complete. There is no concretecertification process now for any manufacturer orsubcontractor. They self-certify in-house processesfrom machining to soldering to software with processeslike ISO9001 and CMMI. Typically, differentcompanies and projects address this challenge andchoose unit tools to perform the upfront analysis andmodeling not following a standard approach. Multipletools need to converge or be compatible with a setmodeling standard for complex systems. The complianttools could be used to verify the requirements up frontto mitigate reliability issues down the life-cycle.A notional approach would follow that shown in Figure10 where the system V is followed but architecturalmodeling and analysis go parallel with the realdevelopment effort. This would allow reliability to bemeasured during the design phase and measured duringthe implementation, test and verification phase using themodel. This approach could bridge the design and testphases together. It is emphasized here that thearchitectural model would not replace critical testing butaugments the process to allow for better requirementidentification and verification. Thorough ground andflight tests should never be replaced by modeling.Modeling would only allow for more robust and ahigher level of confidence in the requirements anddesign. The model could be used in conjunction withthe testing to confirm the design. Proper modeling andanalysis would reduce total program costs by enforcingmore complete and correct initial requirements whichreduce issues discovered down the road in testing thatare expensive to fix or impossible to fix and having toaccept high risks. Additionally, if the model ismaintained and optimized then it could possibly be usedafter system deployment to analyze impacts of upgradesor changes to the system, allowing for more completeanalysis and reduce overall system redesign costs.A hurdle to cross with modeling and analysis isconvincing people to believe those models. Somemethod to certify these models and modeling toolsshould be addressed in the future. Standards should beset in place for correct modeling techniques for complexsystems. Lastly, consideration of standard verificationchecking tools should be made such as with the use ofthe Motor Industry Software Reliability Association(MISRA) compliance verification tool for the use of Cin safety critical systems.SummaryIn conclusion, methods for achieving a design forcomplex systems do exist; however, achievingreliability and attaining a level of qualification thatwould permit better system integrity and basis ofqualification and test does not. It is critical that astandard is developed to tackle this issue rather thanrelying on current methods to ascertain airworthiness ofcomplex systems. This will not occur overnight. If itnot addressed then systems will continue to becomeever increasingly expensive in money and schedule.An orchestrated collaboration among industry,academia, military labs, and technically professionalsocieties to focus on development of this standardshould allow us to draw upon the experiences to feedinto this reliability standard with AWR safety in mind.We have long living existing complex software systemson the Space Transportation System (STS),International Space Station (ISS), missile systems,nuclear submarines and ship systems, nuclear controlsystems, military and commercial jet systems fromwhich we should be able to obtain at least inferredsoftware reliability information from the architectureand run time that the systems use. We should look at thelessons learned from these systems to see what couldhave been done to improve and what was done right thatshould be carried forward. The challenge is collectingthis information into a central database and arriving atsome figure of reliability from previous systems wheredata exists. This would at least provide a starting pointto allow initial assessments and could be optimized inthe future. This would not be an easy task since theinformation is dispersed. Also, historical analyses onfailures are subject error, may be inaccurate orpolitically sensitive. Additionally, there have beenother studies in the past for establishing reliabilitymetrics to complex software systems. Research projectsof similarity to this effort that have come and gone. Ifthose studies had merit then data from those projectsshould not be wasted but studied to feed into whateverstandard that is developed. Even if the result came todead end then future study on the same path should notbe repeated. While historical information would beuseful, each design is unique and requires tools toaccomplish the design. Investigation of architecturalmodeling constructs should be further investigated as apossible augmentation to the design and test process.We need to determine which forum is best to conductthis effort (e.g., SAE, IEEE, AIAA, ACM, AHS,INCOSE, or other). As stated in the paper “SpaceShuttle Avionics” [31], “The designers, the flight crew,and other operational users of the system often have amindset, established in a previous program orexperience which results in a bias against new ordifferent, „unconventional‟ approaches”. If nothing isdone to address this problem it will only get worse overtime. It is past time to address the issue of reliability ofcomplex systems and software.12

.Figure 9 - The Aggregation of Components to Systems Level ReliabilityFigure 10 - Notional System V development curve with Architectural Modeling and AnalysisCoupled13

Acronym ListAcronymAADLACACMAEDAFTDAGCAHSAIAAAMCOMAMRDECARARINCARPASIFATAMATMAWRCAASCH-47CMMCMMICMUCNSCoSMICCPSCRCDFBWDoDE 3ESMLFAAFCSFHAFMECAFTAGPWSIBMIECILDefinitionArchitectural Analysis and DesignLanguageAdvisory Circular (FAA)Association of Computing MachineryAviation Engineering Directorate(AMRDEC)Aviation Flight Test Directorate (USArmy)Apollo Guidance ComputerAmerican Helicopter SocietyAmerican Institute of Aeronautics andAstronautics (Inc.)Aviation and Missile Command (USArmy)Aviation and Missiles Research,Development and Engineering Center(US Army)Army RegulationAeronautical Radio Inc.Aerospace Recommended PracticeAvionics Software Integration FacilityArchitecture Tradeoff AnalysisMethodAir Traffic ManagementAirworthiness ReleaseCommon Avionics ArchitectureSystemCargo Helicopter, ChinookCapability Maturity ModelCapability Maturity Model IndexCarnegie Mellon UniversityCommunications, Navigation,SurveillanceComponent Synthesis using Model-Integrated ComputingCyber-Physical SystemChemical Rubber Company (i.e., CRCPress)Digital Fly-By-WireDepartment of DefenseElectrical and Electromagnetic EffectsEmbedded System ModelingLanguageFederal Aviation AdministrationFuture Combat SystemsFunctional Hazard AssessmentFailure Mode, Effects and CriticalityAnalysisFault Tree AnalysisGround Proximity Warning SystemInternational Business MachinesInternational Engineering ConsortiumInstrumentation Lab (now DraperLaboratory)AcronymIMAINCOSEISOISSKALMISRAMITNASAPDRPEOPNASRAQRMARTCRTTCRTCASAESCADESEDSEESSEISILSSASTSSysMLTCTMRQMRTRLUASUAVUH-60UMLU.S.USLDefinitionIntegrated Modular AvionicsInternational Council on SystemsEngineeringInternational Organization forStandardizationInternational Space StationKorean AirlinesMotor Industry Software ReliabilityAssociationMassachusetts Institute of TechnologyNational Aeronautics and SpaceAdministration (USA)Preliminary Design ReviewProgram Element OfficeProceedings of the National Academyof SciencesRotorcraft and Aircraft QualificationRate Monotonic AnalysisRedstone Test Center (US Army)Redstone Technical Test Center (USArmy)Radio Technical Commission forAeronauticsSociety of Automotive EngineersSafety Critical ApplicationDevelopmentSoftware Engineering Directorate(AMRDEC)Software Engineering EvaluationSystemSoftware Engineering Institute (CMU)System Integration LaboratorySystem Safety AssessmentSpace Transportation SystemSystems Modeling LanguageType CertificateTriple Modular RedundantQuad Modular RedundantTechnical Readiness LevelUnmanned Aircraft SystemUnmanned Air VehicleUtility Helicopter, BlackhawkUnified Modeling LanguageUnited StatesUniversal Systems LanguageReference DocumentsDocument ID Document NameADS-51-HDBK Rotorcraft and Aircraft QualificationHandbookADS-64-SP Airworthiness Requirements forMilitary Rotorcraft14

Document ID Document NameADS-48-PRF Performance Specification forAirworthiness QualificationRequirements for Operation ofAircraft in IMC and Civil InstrumentFlight RulesADS-75-SS Army Aviation System SafetyAssessments and AnalysesAMCOM-385-17 Safety StandardAR-70-62 Airworthiness Qualification ofAircraft SystemsAR-95-1 Aviation Flight RegulationsARINC 653 Specification Standard for Time andSystem PartitionDO-178B Software Considerations in AirborneSystems and Equipment CertificationDO-248B Final Report for the Clarification ofDO-178BDO-254 Design Assurance Guidance forAirborne Electronic HardwareDO-278 Guidelines for Communications,Navigation, Surveillance, and AirTraffic Management (CNS/ATM)Systems Software Integrity AssuranceDO-297 Integrated Modular Avionics (IMA)Development Guidance andCertification ConsiderationsFAA-AC-25.1309Equipment, Systems, and Installationsin Part 23 AirplanesFAA-AC-27-1B Advisory Circular - Certification ofNormal Category RotorcraftFAA-AC-29-2C Advisory Circular - Certification ofTransport Category RotorcraftISO/IEC-12207 Software Life-cycle ProcessesMIL-STD-461 Requirements for the Control ofElectromagnetic InterferenceCharacteristics ofSubsystems and EquipmentMIL-STD-810 Department of Defense Test MethodStandard for EnvironmentalEngineering Considerations andLaboratory TestsMIL-STD-882D Department of Defense System SafetyPMHFSA-001 SED SEES Program ManagerHandbook for Flight SoftwareAirworthinessPMHSS-001SED SEES Program ManagerHandbook for Software SafetySAE-ARP-4754 Certification Consideration for HighlyIntegrated or Complex AircraftSystemsSAE-ARP-4671 Guidelinesand Methods for Conducting theSafety Assessment Process onAirborne Systems and EquipmentBibliography[1] Israel Koren and Mani Krishna, “Fault-TolerantSystems”, Morgan Kaufmann, 2007[2] Jianto Pan, “Software Reliability”, Carnegie MellonUniversity, Spring 1999[3] Nachum Dershowitz, http://www.cs.tau.ac.il/~nachumd/horror.html[4] http://www.air-attack.com[5] David A. Mindell, “Digital Apollo: Human andMachine in Spaceflight”, The MIT Press, 2008[6] Software Engineering Directorate: SoftwareEngineering Evaluation System (SEES), “ProgramManager Handbook for Flight Software Airworthiness,SED-SES-PMHFSA 001, December 2003[7] Software Engineering Directorate: SoftwareEngineering Evaluation System (SEES), “ProgramManager Handbook for Software Safety, SED-SES-PMHSSA 001, February 2006[8] AMCOM Regulation 385-17 Software SystemSafety Policy, 15 March 2008[9] NASA Software Safety Guidebook, NASA-GB-8719.13, 31 March 2004[10] Margaret Hamilton & William Hackler, “UniversalSystems Language: Lessons Learned from Apollo”,IEEE Computer, IEEE Society, December 2008,http://www.htius.com/Articles/r12ham.pdf[11] M. Hamilton, "001: A Full Life Cycle SystemsEngineering and Software Development Environment;Development Before the Fact in Action", cover story,special editorial supplement, 22ES-30ES, ElectronicDesign, June 1994.http://www.htius.com/Articles/Full_Life_Cycle.htm[12] Peter Feiler, David Gluch, John Hudak, “TheArchitecture Analysis & Design Language (AADL): AnIntroduction”, CMU/SEI-2006-TN-011, February 2006[13] Peter Feiler, John Hudak, “Developing AADLModels for Control Systems: A Practitioner‟s Guide”,CMU/SEI-2007-TR-014, July 2007[14] Bruce Lewis, “Using the Architecture Analysis andDesign Language for System Verification andValidation”, SEI Presentation, 2006[15] Feiler, Gluch, Hudak, Lewis, “Embedded SystemArchitecture Analysis Using SAE AADL”, CMU/SEI-2004-TN-004, June 2004[16] Charles Pecheur, Stacy Nelson, “V&V ofAdvanced Systems at NASA”, NASA/CR-2002-211402, April 2002[17] Systems Integration Requirements Task Group,“ARP 4754: Certification Considerations for Highly-Integrated or Complex Aircraft Systems”, SAEAerospace, 10 April 1996[18] SAE, “ARP 4761: Guidelines and Methods forConducting the Safety Assessment Process on CivilAirborne System and Equipment”, December 1996[19] Aeronautical Radio Inc. (ARINC), “ARINCSpecification 653P1-2: Avionics Application Software15

Standard Interface Part 1 – Required Services”, 7 March2006[20] Department of Defense, “MIL-STD-882D:Standard Practice for System Safety”, 19 January 1993[21] RTCA, Incorporated, “DO-178: SoftwareConsiderations in Airborne Systems and EquipmentCertification”, 1 December 1992[22] RTCA, Incorporated, “DO-254: Design AssuranceGuidance for Airborne Electronic Hardware”, 19 April2000[23] US Army, “Aeronautical Design StandardHandbook: Rotorcraft and Aircraft Qualification(RAQ) Handbook”, 21 October 1996[24] Cary R. Spitzer (Editor), “Avionics: Elements,Software and Functions”, CRC Press, 2007[25] US Army, “Army Regulation 70-62: AirworthinessQualification of Aircraft Systems”, 21 May 2007[26] US Army, “Army Regulation 95-1: Aviation FlightRegulations”, 3 February 2006[27] Barbacci, Clements, Lattanze, Northrop, Wood,“Using the Architecture Tradeoff Analysis Method(ATAM) to Evaluate the Software Architecture for aProduct Line of Avionics Systems: A Case Study”, July2003, CMU/SEI-2003-TN-012[28] “All in the Family: CAAS & AADL”, Peter Feiler,August 2008, CMU/SEI-2008-SR-021[29] Chrissis, Konrad, Shrum, “CMMI: Guidelines forProcess Integration and Product Improvement”, PearsonEducation, 2007[30] Brendan O‟Connell, “Achieving Fault Tolerancevia Robust Partitioning and N-Modular Redundancy”,Draper Laboratory, January 2006[31] John F. Hanaway, Robert W. Moorehead, “SpaceShuttle Avionics Systems”, NASA SP-504, 1989[32] Lui Sha, “The Complexity Challenge in ModernAvionics Software”, August 14, 2006.[33] “Incidents Prompt New Scrutiny of AirplaneSoftware Glitches”, 30 May 2006, Wall Street Journal[34] Eyal Ophir, Clifford Nass, and Anthony Wagner,“Cognitive Control in Media Multitaskers”, PNAS, 20July 2009[35] “Advisory Circular AC 25.1309-1A: SystemDesign and Analysis”, Federal Aviation Administration,21 June 1988[36] Program Element Office (PEO) PolicyMemorandum 08-03 (Risk Matrix)[37] Nancy Leveson, “The Role of Software inSpacecraft Accidents”, MIT, AIAA Journal ofSpacecraft and Rockets[38] http://www.nsf.gov/pubs/2008/nsf08611/nsf08611.htm, National Science Foundation webpageon Cyber-Physical Systems.[39] Hoang Pham, “Software Reliability”, Springer,2000[40] Paul Rook, editor, “Software ReliabilityHandbook”, Elsevier Science Publishers, 1990[41]http://www.navair.navy.mil/v22/index.cfm?fuseaction=news.detail&id=128[42]http://mars8.jpl.nasa.gov/msp98/news/mco990930.html[43]John Garman, “The Bug Heard Around the World”,ACM SIGSOFT, October 1981[44]http://marsprogram.jpl.nasa.gov/MPF/newspio/mpf/status/pf970715.html, “Mars Pathfinder MissionStatus”, July 15, 1997[45] Nancy Leveson, “Safeware: System Safety andComputers”, Addison-Wesley Publishing Company,1995[46] http://www.electronicaviation.com/aircraft/JAS-39_Gripen/810[47] Brandon Hill,http://www.freerepublic.com/focus/fnews/1791574/posts,Lockheed's F-22 Raptor GetsZapped by International Date Line, DailyTech LLC ,February 26, 2007[48] http://www.military.com/news/article/human-errorcited-in-most-uav-crashes.html[49] Daniel Michaels and Andy Pasztor, “IncidentsPrompt New Scrutiny Of Airplane Software Glitches,As Programs Grow Complex, Bugs Are Hard to Detect;A Jet's Roller-Coaster Ride, Teaching Pilots to GetControl” Wall-Street Journal, May 30, 2006AcknowledgementsIn addition to the references, the authors would like toexpress their appreciation to those who reviewed andprovided critical input through discussion for this paper.Those people included: Dr. William Lewis, DavidCripps, Kevin Rotenberger, Fred Reed, Yen Fuqua,Michael Walsh Wayne Alan Garrison, Ward Saemisch,Larry Fisher, John Byerly, Dan Francis, Glenn Carter,Raquel Gardner, Adam Robinson, Mazen Khazendar,Michael Dreyer, Bruce Lewis, Josh McNeil, MelvinJohnson, Steven Hosner, Michael Kidd, RandyCrockett, Margaret Hamilton, Dr. Paul Miner, SusanDeGuzman, David Homan, Dr. Jose Lopez, MichaelAucoin, Roger Racine, Gary Schwarz Mervin Brokke,Amy Boydston and the nice folks at Redstone ScientificInformation Center (RSIC). It should be noted that theviews of the reviewers does not necessarily agree withthe author(s).About the AuthorsAlex Boydston has over 17 years of experience inDepartment of Defense, NASA and commercialtelecommunication systems, test, and softwareengineering. He has held engineering positions atTeledyne Brown Engineering, SCI, ADTRAN, DraperLaboratory, and is now employed with the United StatesArmy Aviation Engineering Directorate all positionswithin Huntsville, AL. From 1990 to 1992 Alex was acooperative education student in Advanced SpacePrograms with Teledyne Brown Engineering working16

Space Station Furnace Facility and Protein CrystalGrowth Facility. From 1992 to 1994 he was aCommunications and Systems Test Engineer forTeledyne Brown Engineering on the National MissileDefense program. From 1994 to 1996 Alex was thelead systems integrator of experiments and hardware forthe Space Shuttle Middeck Glovebox. In 1996 heworked as a Tactical Systems Electronics Engineer forSCI, Inc working on the audio systems for the F-16 andF-18 aircraft. From 1996 to 2007 he worked atADTRAN in various roles of Technical Support (2years), Product Qualification Test (5 years), andEmbedded Software Design Engineering (4 years) forenterprise and carrier grade data and voicecommunication systems. From 2007 to 2008 he servedas a lead Flight Processing Architect for theConstellation Ares I rocket for Draper Laboratory atMarshall Space Flight Center. In 2009 Alex beganworking for the U.S. Army as an Electronics Engineerin the Avionics Branch of Mission Equipment for theAviation Engineering Directorate. Mr. Boydston holdsa Bachelor of Science (B.S) Degree in ElectricalEngineering (1992) and Masters of Science (M.S.)Degree in Electrical Engineering (2001), both from theUniversity of Alabama in Huntsville.Dr. William D. Lewis was selected for the SeniorExecutive Service in July 2006. As the Director ofAviation Engineering, he is the Airworthiness authorityfor Army aircraft and provides matrix support tocustomers. Dr. Lewis‟ direct customers are the ProgramExecutive Officer Aviation Program/Project/ ProductManagers. The ultimate customers are the Army aircraftcrew, passengers, and maintainers that operate the Armyaviation systems. In this capacity, he insures flightsafety of all Army aircraft via design and flightrestrictions. Additionally, he is responsible for thedesign and performance verification of aircraft andequipment to support Army aviation. From 2004 topresent Dr. Lewis is the Acting Director, AviationEngineering Directorate, U.S. Army Aviation andMissile Research, Development, and EngineeringCenter, Redstone Arsenal, AL. In 2004 he was BranchChief, Flight Controls, Aeromechanics Division,Aviation Engineering Directorate, U.S. Army Aviationand Missile Research, Development, and EngineeringCenter, Redstone Arsenal, AL. From 2003 to 2004 hewas a Chief Engineer, RAH-66 Comanche ProgramU.S. Army Aviation and Missile Research,Development, and Engineering Center, RedstoneArsenal, AL. From 2002 to 2003 he was a SystemsEngineer, Comanche Program, U.S. Army Aviation andMissile Research, Development, and EngineeringCenter, Redstone Arsenal, AL. From 1996 to 2002 hewas Professor at University of Tennessee, Tullahoma,TN. From 1992 to 1996 he was a Product Manager withthe US Army Aviation Systems Command, Ft Eustis,VA (Retired from Army after 21 years of service at rankof LTC). From 1989 to 1992 he was a SimulationEngineer in Atlanta, GA. From 1986 to 1989 he was anExperimental Test Pilot, US Army Aviation SystemsCommand, Edwards Air Force Base, CA. From 1983 to1985 Dr. Lewis was an Aerospace Engineer with the USArmy Aviation Systems Command, St. Louis, MO. Dr.Lewis holds a Ph.D. in Aerospace Engineering fromGeorgia Institute of Technology, an M.S. in AviationManagement, Embry-Riddle Aeronautical Universityand a B.S. from the United States Military Academy atWest Point.DISCLAIMERPresented at the AHS Specialists’ Meetingon Systems Engineering Harford, CT onOctober 15-16, 2009. This material isdeclared a work of the U.S. Governmentand is not subject of copyrightmaterials. Approved for public release;distribution unlimited. Review completedby the AMRDEC Public Affairs Office 21Sep 2009; FN4208. Reference herein toany specific commercial, private orpublic products, process, or service bytrade name, trademark, manufacturer, orotherwise, does not constitute or implyits endorsement, recommendation, orfavoring by the United States Government.The views and opinions expressed hereinare strictly those of the authors and donot represent or reflect those of theUnited States Government.17