"Experiences in Applying Testing Automation Research in an ...

"Experiences in Applying TestingAutomation Research in anIndustrial Context."Marlon Vieira, PhDProgram Manager - Software Development TechnologiesSoftware Engineering DepartmentSiemens Corporate ResearchPrinceton, NJEmail: marlon.vieira@siemens.comCopyright © Siemens AG 2008. All rights reserved.

Outline1. Background2. Some aspects from Research toTechnology Transfer3. A Broader View: Software Engineering,Testing and Automation4. Experiences from applying our MBT Research inseveral projects5. Typical Aspects Used to Evaluate Introduction ofnew Technologies and Final RemarksPage 2Siemens Corporate Research

Siemens has a long tradition oftechnological innovations.Page 3Siemens Corporate Research

Siemens is one of the world’s largestsoftware companies• Siemens has more than 41,000 software developers.• 60% of Siemens’ business is based on software.• Siemens spends more than 3 billion Euros per year onsoftware development.Page 4Siemens Corporate Research

But, Siemens is not recognized as a softwarecompany, since most of our software isembedded.Examples:• Automation devices• Industrial control systems• Medical devices• Communication systems• Rail systems• Automotive componentsPage 5Siemens Corporate Research

Corporate Technology:About 2,300 Researchers and Developers WorldwideRomsey (RMR)BerlinSt. PetersburgMoscowPrincetonSiemensCorporate TechnologyA Global NetworkBeijingBerkeleyShanghaiTokyoErlangenMunichBangalorePage 6Siemens Corporate Research

Global Technology FieldSoftware Development TechnologiesPage 7Siemens Corporate Research

From Research to Practice“Genius is one percent inspiration andninety-nine percent perspiration.”- Thomas EdisonFrom Research to Prototypes & Solutions TransferPre-development• Research requirements• Technology evaluation• Academic partners• Feasibility study• PrototypePrototype/Work-in-Progress• Prototypeoptimization• Pilots• Feedback• ValidationTransfer• Integration withproducts and/orprojects• More CompleteDocumentation andknowledge transferProductization &Maintenance• Maintenance• UpgradesPage 8Siemens Corporate Research

Typical Characteristics in Research that weapply in practices• Scalability• Usability• Early results• Support Evolution and,• Different levels of abstractionPage 9Siemens Corporate Research

Broader View“Software Challenges at Industrial Context”• Functionality previously realized inelectrical or electro-mechanicalsystems is now being realized insoftware• Meeting functional and non-functionalrequirements is important to businesssuccess• Multisite development projects.• High quality (i.e., thoroughly tested,reliable) software is important tobusiness success.Page 10Siemens Corporate Research

Software Testing: Some challenges• Pressure to release software to the market(with quality and on time)• Interoperability Testing• need to interact with other constituents(Siemens products and/or from othercompanies)• Specification is a Moving Target• new customer requirements•Testing is Expensive• laboratory testing vs. field testingPage 11Siemens Corporate Research

An Essential challengeEffective ways to design and automate library of teststhat contains the right subset of possible test caseshaving the highest probability of detecting errorsPage 12Siemens Corporate Research

Typical testing processTest Planning Test Design Test Execution Test ReportingTest Planning Test Design Test Execution Test ReportingPage 13Siemens Corporate Research

Automation of Testing:Observations (Diverse Siemens Projects)The test process and its automation in practice (Tools)Test Management ToolsCapture-ReplaySimulators/emulatorsCustom FrameworksResult Analysis ToolsDefect ClassificationMetrics AnalysisUnitComponent Integration ValidationHigh * Medium * Medium * Low *Page 14* Automation level (usage)Siemens Corporate Research

Common FrameworkExistence of different frameworks for different steps of the test processtest scriptsGUISUTTest resultsobjectTest ManagementdatabaseToolPage 15Siemens Corporate Research

Automation and Test Design• Test Design is usually the weakest link• Use a test management tool for what belongs in it• Test Design should be leveraged into AutomationPage 1616Siemens Corporate Research

Model-Based Testing: Our View“ MBT is a technology and not a solution by itself ”• Automatic generation of test cases using models createdduring development process or extracted (created) fromartifacts of that process.• Average testers are already using models.• Testing creation activities are based (at least) on mentalmodelPage 18Siemens Corporate Research

MBT ProcessModeling Generating Executing Analyzing ResultsMentalModelGenerateTest CasesTest SuiteTest SuiteRequirementor DesignDocumentationDecide whether to• Generate more tests• Modify the model• Stop testingTest passor FailOraclesVerifyResultsand/orGenerateExecutableCreateExecutableExecutableExecutableScriptsScriptsRunScriptsExecuteManuallyApplication Under TestPage 19Siemens Corporate Research

MBT Research at Siemens Corporate Research• Workbench for developing efficienttechniques for test case generation• Main goal - Support an industrial and“affordable“ approach• Increased use of model-based development (UML) bySiemens Units• Leverage testing by reusing documents/models• Opportunity for automatic test case generationPage 20Siemens Corporate Research

TDE/UML UC-Activ : Main Concepts• Support a Functional (Sub) System (black box) testing• Integration to different tools to support the Test Methodology• Methodology combines:Modeling system behavior− UML Modeling tools / SCR Eclipse UML Editor− UML UC and Activity Diagrams− Extension (via Stereotypes) to better describe testablespecificationsGenerating test cases− Diverse control/data coverage criterion− Test Management ToolsExecuting test cases− Approach to create executable test cases− Test Execution ToolsPage 21Siemens Corporate Research

MBT with TDE/UMLModeling• Focus on UML Use CaseDiagrams refined by UMLActivity DiagramsTest Case Generation• Combines Graph Coverage andData Coverage to form a TestSuiteCode Generation• Transforms a Test Suite intoother formats, such asexecutable code, reports, xml…Page 22Siemens Corporate Research

Modeling for TDE/UMLAll Use Cases must have an Activity Diagram that describes itUse Cases have a network of included and extended use casesUse Cases can be included and extended by many use casesThis promotes reusePage 23Siemens Corporate Research

Modeling for TDE/UML• Categories and Choices• Not part of the UML• Refers to the Category-Partition method developedwithin SCR - Thomas J. Ostrand and Marc J.Balcer[CACM ,1988].• Categories:• The main idea is to partition the input domain offunction being tested, and then select test data for eachclass of the partition• A category is defined by specifying all possible datachoices that it can represent• Choices:• Data values of a category• Within a category, choices are required to be of thesame typePage 24Siemens Corporate Research

Test Case GenerationData Coverage Criterion• Sampling• Choice-per-suite• Choice-per-variable• Choice-per-path• Exhaustive2 Tests, 48 ProceduresPage 25Siemens Corporate Research

TDE/UML - Some Publication• “Automation of GUI Testing Using a Model-driven Approach”, M. Vieira, J. Leduc, B. Hasling, R.Subramanyan, J. Kazmeier Workshop on Automation of Software Test (AST'06). At 28th InternationalConference on Software Engineering (ICSE'06), Shanghai, China.• “An approach for generating and executing tests based on the UML testing profile”, M. Vieira. InProceedings of the IASTED International Conference on Software Engineering, February, 2006 Innsbruck,Austria• "A UML-based approach to system testing". Published at Innovations in Systems and SoftwareEngineering (A NASA Journal), Hartmann J, Vieira M., and Ruder A.. Issue: Volume 1, Number 1 , ISSN:1614-5046 Publisher: Springer-Verlag London Ltd DOI: 10.1007/s11334-005-0006-0, Pages: 12 - 24 , April2005• "TDE/UML: A UML-based Test Generator to Support System Testing", Hartmann J, Vieira M., andRuder A.. In Proceedings of the QAI's 5th International Software Testing Conference, February, 2005,Hyderabad, India• The Category-Partition Method for Specifying and Generating Functional Tests, T. J. Ostrand, Marc J.Balcer, CACM 31(6): 676-686 (1988)• “Applying Model-Based Testing to Healthcare Products: Preliminary Experiences” Marlon Vieira,Xiping Song, Gilberto Matos, Stephan Storck, Rajanikanth Tanikella, Bill Hasling --ICSE 2008– HealthcareSessionPage 26Siemens Corporate Research

Practical ApplicationRequirements+horizontal workflowsvertical, screen-based workflowsPage 27Siemens Corporate Research

Process OverviewPage 28Siemens Corporate Research

Some Results• Recognition that Incorporation of MBT reducecomplexity and increase productivity• “It took about 5 minutes to update the model for the newrequirement. 3 minutes to generate the tests and 2 minutes toexport the tests to Quality Center. What a time saver!” –TestAnalyst• Time to address bigger test issues• Decreased the number of escaped defects• Improved tester job satisfactionPage 29Siemens Corporate Research

Some Challenges in Applying MBT (Supportedby TDE/UML) in an Industrial Project• Comfort factor• Difficult to convince of benefits initially• Process shift• Up front investment in test• Personnel shift• Test designers with stronger abstraction skills• Automation of test execution• Not easy effortPage 30Siemens Corporate Research

TDE/UML From Research to PracticeFrom Research to Prototypes & Solutions TransferPre-development• Research requirements• Technology evaluation• Academic partners• Feasibility study• PrototypePrototype/Work-in-Progress• Prototypeoptimization• Pilots• Feedback• ValidationTransfer• Integration withprojects (process)• More CompleteDocumentation andknowledge transferProductization &Maintenance• Maintenance• UpgradesPage 31Siemens Corporate Research

Typical Aspects Used to EvaluateIntroduction of new TechnologiesDefine goals & process; then consider costs:1. Analysis goals, requirements - level of dissatisfaction withthe status-quo (compelling event)2. Selection: Initial (internal) knowledge acquisition orpurchase or partnership3. Pilot4. Installation & Rollout5. Training6. Maintenance7. Evangelizing, internal selling8. Organizational process changesWeigh the cost (each of the steps) against the benefit thatimplementing this new technology will bring, is it worth it?Page 32Siemens Corporate Research

Final Remarks (1)"Experiences in Applying Testing Automation Researchin an Industrial Context." Introducing new Technologies is not easy.• Change must be positive and well managed, with aminimum of disruption and culture shock. Determine readiness for change• level of dissatisfaction with the status-quo (compellingevent)versus• cost of change (resistance)Page 33Siemens Corporate Research

Final Remarks (2)"Experiences in Applying Testing Automation Researchin an Industrial Context." Justify the improvements; define the benefits• Sell the change; gain consensus– From senior management to practitioner– From practitioner to senior managementShow value added to management, developers andrest of organization• Measure the effectivenessPage 34Siemens Corporate Research

Thank you for your attention!Marlon Vieiramarlon.vieira@siemens.comPhone +1 609. 2332300Page 35For internal use only / Copyright © Siemens AG 2007. All rights reserved.Siemens Corporate Research