Davidson Warfighter Inclusion in System Development final R1

1
Dr. Daniel Wallace, NAVSEA Human Factors Engineering Technical Warrant Holder;
Mr. Dennis White, NAVSEA Manpower and Personnel Technical Warrant Holder;
Ms. Karole Davidson, NSWCDD Human Systems Integration Program Director
WARFIGHTER INCLUSION IN SYSTEM DEVELOPMENT:
THE OPERATIONAL PERSPECTIVE IN DEFINING DESIGN REQUIREMENTS
ABSTRACT
Human Systems Integration (HSI) is integral to any comprehensive system engineering process.
HSI is defined by the International Conference on System Engineering as an “…
interdisciplinary technical and management processes for integrating human considerations
within and across all system elements; an essential enabler to systems engineering practice." But
how does one accomplish this interdisciplinary integration? This paper highlights the essential
role of the “Warfighter” as an overarching elicitor and integrator of system requirements, and
how to effectively exploit this crucial resource.
All complex systems are developed to be used and maintained in a specific operational
environment by a trained user. It is therefore imperative that the unique operational perspectives
of those users are made part of the design and development process. There are several key
opportunities within the acquisition process where qualified, representative users play a
significant role in refining the systems’ requirements and defining how the total system will
perform. The active participation of the Warfighters and user community as part of the design
process identifies issues early enough to make a meaningful influence on design, fosters
Warfighter acceptance of the system, reduces the risks associated with Warfighter alterations to
system configurations, and validates total system performance using human in the loop
evaluations.
BACKGROUND AND PROBLEM
Human Systems Integration (HSI) is integral to any comprehensive system engineering process.
HSI is defined by the International Conference on System Engineering (INCOSE) as an “…
interdisciplinary technical and management processes for integrating human considerations
within and across all system elements; an essential enabler to systems engineering practice."

But how does one accomplish this interdisciplinary integration, and why is it even necessary?
With the challenges facing program managers (PMs) to meet cost and schedule requirements, the
motivation is often to lock down requirements as early as possible. Unfortunately, however, this
restricts the opportunities for the operational experts (the Warfighters) to effectively influence
the design as it matures. The Warfighter cannot always provide a meaningful review of a system
based on the requirements as specified in a CDD or other requirements documents; rather, they
need to see an evolving conceptual design and ases it’s suitability within their operational
mindset.
DIFFERENT PERSPECTIVES OF WARFIGHTERS AND ENGINEERS
There is a fundamental difference in the perspectives of the Warfighter community and typical
engineers and system developers. This is an easy truism to which one can pay lip service, but it
is rarely appreciated by the engineers. We think we understand all of the system design (or
operational) requirements, because we understand the capability (or performance) requirements,
but these are very different entities.
This is best illustrated by an example. In looking at the detect-to-engage sequence in an air
defense scenario, the system capability requirements are well understood: 1) Detect an air threat
of type X at range Y; 2) Determine the optimal weapon-target pairing; and 3) Determine best
time to fire salvo to ensure maximum Pk. And if you were to ask an engineer what the critical
decisions were in this scenario, he would likely say, “What is the best weapon to use, and when
should I fire it to achieve the mission success?” And this would be entirely consistent with the
system capabilities perspective. However, the Warfighter would have an entirely different
response. To the Warfighter, the critical decisions are, “What is the track (ID)?”, “How sure am
I of that ID?”, and “What are the consequences if I am wrong?”This is the operational
perspective.
Clearly, the difference between these perspectives has profound implications for the systems
engineering and design of the resulting system. The information displayed for the Warfighter
must support the operational perspective, and these design requirements can only be captured via
Warfighter input.
APPROACH TO A SOLUTION
Including Warfighters as part of the Human System Integration (HSI) plan early in design and
leveraging their operational perspective is helping to bring low cost improvements and solutions
into the Navy. Figure 1 depicts various activities and/or products where human system engineers
participate in the system engineering process and in each phase the Warfighter can, and should,
play a central role.
2

3
HSI in the Systems
Engineering Process
Identify Human Performance Component
of Total System Capability/Reqmts.
Identify HSI Research Needs
MNS/ORD/SSS Review for HSI Impact
Identify Quality of Life Issues
Discern HSI Metric Development Needs
Mission Need
Document all Functions/Tasks
Identify Human / Automation Allocations
Evaluate Technology Integration Feasibility
Analysis of Environmental Conditions
Identify Decision Support Requirements
Requirements
Analysis
Requirements Loop
Verification
Functional
Analysis/
Allocation
Design Loop
System Analysis
& Control
Synthesis
System
Capability
FUNCTIONS
Manning Optimization
Human Performance Optimization
Workspace / Berthing Design
Decision Support Aiding
Usability Engineering
HMI Design
T&E Support
Metric Development
TOOLS
Modeling and Simulation Software
Operator Workload Analysis
Cognitive Task Analysis
Prototype Development
Usability Testing
Knowledge Management Tools
DOD and Industry HSI Pubs
TRADE STUDY CRITERIA
•Human Performance
•Usability
•Maintainability
•Cost
•HMI
•Quality of Life
•Redundancy
Participate in IPT’s to
ensure optimization of the
Total System by designing to
optimize the Human Element
Figure 1. System Engineering Process with Human System Integration
A system needs to be defined as the hardware, software, and the people who operate, maintain,
and make decisions using it. Total system performance is the intersection of all these pieces of
the system –simply describing performance parameters for the hardware and software miss the
true intent of the use of the system by the operators/maintainers, which must be part of the
calculation. As part of the requirements analysis, a critical decision analysis is important –since
the engineer’s view is quite diferent from the view of the warfighter, as cited above. Therefore,
the warfighter must be involved to validate that the design will indeed support his/her mission.
EARLY WARFIGHTER INVOLVEMENT IN FUNCTIONAL ANALYSIS
One essential part of the system engineering process is the traditional functional task analysis, or
Top-Down Functional Analysis (TDFA). The TDFA identifies candidate functions or tasks for
automation, elimination, consolidation, or simplification based upon analysis. The analysis
investigates which tasks should be performed by the warfighters versus the hardware (equipment)
or software (computer program/automation). “It is the human element that provides us the
flexibility and creativity to deal with the unanticipated threat in the unexpected situation that
prevents the enemy from being able to predict preprogrammed responses.” (Leading Edge, Vol.

6, Issue 1, pg 8) For example, people are not very good at watching a screen to determine when
something changes (vigilance). In this case, the software system should alert the operator.
People are, however, great at putting together a pattern from divergent pieces of data. Inclusion
of human systems integration as part of the system engineering process reflects the operational
perspective and allows a focus on the interaction of the user with the other elements of the
system where consideration is given to human capabilities and limitations (perceptual, cognitive,
and physical attributes). Some of the benefits of incorporating the user into the system design
process include designing systems that are optimal for human use and enhance performance by
reducing error and increasing productivity, enhancing safety and comfort, increasing levels of
satisfaction, and increasing the warfighter’s ability to learn quickly.
WARFIGHTER INVOLVEMENT IN USABILITY ENGINEERING
A key element in warfighter-assisted design is in iterative usability engineering and assessments.
System design concepts should be exposed to the user community very early in design –there’s
no need to wait until the system is developed –to ensure that these early concepts conform to an
operational model. Early usability engineering can be simply and economically performed with
PowerPoint and user cognitive walk-throughs, and later progress to more functional physical or
software mockups/prototypes. Another element is trade studies, where specific design questions
are answered with a focused test. Warfighters participate in the various test events in order to
measure their actual performance. The goal is not to simply provide their subjective opinion but
to objectively measure human performance. Using operationally relevant scenarios with trained
and qualified Warfighters to perform the envisioned system’skey requirements can be verified
early in the design process when the cost to make changes to the concept or design is minimal.
NEED FOR BALANCE BETWEEN OPERATIONAL AND TECHNICAL CAPABILITY
PERSPECTIVES
Completing the engineering team by including the warfighter is a great benefit, however, the
goal is to have a warfighter from the fleet involved periodically throughout the various stages of
development. A warfighter or former warfighter assigned to the program and constantly exposed
to engineers and the program of interest can heavily influence the way a sailor or marine looks at
the environment; engineers can corrupt a warfighter if they are not careful. The warfighter with
a fresh perspective can provide feedback both on the design under evaluation as well as
potentially new ideas.
On the flip side, one must be careful to make sure that the operational perspective is not confused
with “the way we do things today”. It would be a step backwards to simply give the warfighters
a new tool that replicates an old way of doing business; to stay ahead of our adversaries; we must
always push for the greatest technological advantage, so long as we keep the operational
4

perspective. This requires a balanced view that captures operational goals and perspectives
while wedding the latest technological capabilities. This is true systems engineering.
DIFFERENT PERSPECTIVES BETWEEN WARFIGHTERS
One additional consideration is that one or two warfighters cannot adequately capture the full
scope of the operational perspective. Design solutions based upon a small sample of warfighters
will produce a highly idiosyncratic display. A good rule of thumb is to iterate design concepts
early and often with about four or five warfighters in each iteration. A few warfighters may
cary over from one iteration to another, so long as “new blood” is brought in with each design
cycle.
EXAMPLES
Further sections of this paper address examples of studies and exercises where human
performance data has been collected and how reflecting the operational perspective had an
impact on system designs.
WATCH TURNOVER
The first example examines a study designed to measure the situation awareness (SA) of an
operator during a typical watch turnover. Two versions were compared: the traditional face-toface
watch turnover and an automated turnover software prototype. In order to collect the
required data, the watch turnover was scripted in order to ensure that the operator received the
same data no matter whether it was the manual process (speaking with other operators) or the
automated proces (data was provided on the operator’s console).Both the verbal script and the
automated process were developed using representative warfighters to ensure that the
information requirements mapped to the operational requirements of the watchstanders. The
situation awareness measures looked at all three levels of situation awareness:
Perception - Do I see/hear/feel it?
Comprehension –Do I understand what I perceive?
Projection –Do I understand the future impacts?
Figure 2 provides the high level results of the experiment where we found that despite a
substantial preference for the automated turnover, objective measures showed no significant
difference in actual performance.
5

6
Measure
Performance Measures
Objective Scores
Subjective Self-Rating:
Understanding (of 100)
Manual
88.0%
57.4
Automated
87.2%
71.8
No statistically significant difference in
performance
Sailors thought they performed better with
automation, but ….
results show that they did not.
We must measure performance.
Figure 2. Example of Why to Measure Performance
This is just one example of what we often see –the opinion of a person on their performance can
be influenced by many factors other than their actual performance. Many of us, for example,
think we will perform better when a new piece of technology is provided to us; however, often
that is not the case and sometimes we even perform worse. When the watchstanders were
objectively measured in this case, it was discovered that they had essentially the same
performance, but they subjectively felt that the technology was helping them. The real answer is
a mix of both the objective and the subjective, and in reality measuring both and weaving them
together provides the fullest answer.
Poor performance can be due to a number of reasons, including poor design, a lack of
understanding of the technology, or insufficient training of the user. Sometimes the technology
just isn’t a solution for the real problem area. BEFORE we invest heavily in technology or
automation, we need to KNOW that it can provide the expected result. In the case above, there
is no performance gain, but there may be a workload or manning reduction savings or a savings
in the length of time it takes to accomplish the tasks. It is important that we use quantitative
metrics - and not just opinions - to guide decisions on where to spend precious DoD dollars.

CONCEPT OF OPERATIONS EXERCISES (COOPEXS)
The next example deals with a variety of concepts of operation exercises conducted by a joint
government and industry team for a new ship design. The Navy (government) technical team in
conjunction with the industry design agent conducted a Concept of Operations exercise
(COOPEX) in January 2005. The program had developed a robust visual prototype of the bridge
(operationally manned space) which had been reviewed by warfighters; however, the team
decided to proceed with a low cost physical mock-up of the space. The experiment involved
three typical operational scenarios –underway replenishment, harbor transit and formation
steaming. The mock-up was a low-cost full-scale version of the bridge made from form-core and
duct tape, designed to only look at some basic issues. The experiment was critical since the
bridge design is radically different than other ship classes that exist today. The warfighters
involved were all qualified to stand watch on existing surface ships’ bridges today, and reflected
the operational perspective. The results of the experiment identified a number of design issues
with some of them having impact on the structural design of the bridge. The experiment
provided a cost avoidance many, many times larger than the cost of the relatively simple mockup
costs. The success of this experiment led to another mock-up of the helicopter control station
which identified multiple usability issues, an egress solution for a safety issue as well as just
simple errors in the ship drawings.
SUMMARY
As demonstrated by the various examples provided as well as other Navy experience, warfighter
interaction allows the Systems Engineering Team to elicit operational requirements and measure
performance related to: physical layout, communication requirements, team composition and
arrangement, conceptual functional and information requirements, display information
requirements and decision support aids, access and egress, perceived procedural, cultural, and
policy issues associated with concepts, and manning concepts as applied to various conditions of
readiness and special evolutions. This interaction is not simply user opinion, but a measured,
formal elicitation of both opinion and human performance metrics conducted and analyzed by
HSI engineers.
The bottom line is best summed up by NAVSEA’s Systems Engineering and Human Systems
Integration Director, Ms. Trish Hamburger, “The real key for us is that human systems
integration in all of its dimensions is a key part of systems engineering, and it has to be
embedded in it at the very beginning, during the analysis of alternatives, and the initial concepts,
and the initial requirements, and then measured throughout the process in order to ensure that our
systems are designed to optimize warfighter performance but, more importantly, to optimize total
system performance to actually meet the mission capability. And the warfighting community –
for us, our Sailors and Marines –is an esential part of making that happen.” In conclusion, by
7

employing a total systems engineering approach that involves not only the hardware and
software but also the people, total systems performance can be achieved.
8

Dr. Daniel Wallace
Naval Surface Warfare Center Dahlgren Division
18444 Frontage Road
Suite 327
Dahlgren, VA 22448-5161
Phone 540-653-8097
Fax 540-653-2514
Daniel.f.wallace@navy.mil
Mr. Dennis White
Naval Surface Warfare Center Dahlgren Division
18444 Frontage Road
Suite 327
Dahlgren, VA 22448-5161
Phone 540-653-2133
Fax 540-653-2514
Dennis.white3@navy.mil
Ms. Karole Davidson
Naval Surface Warfare Center Dahlgren Division
18444 Frontage Road
Suite 327
Dahlgren, VA 22448-5161
Phone 540-653-1241
Fax 540-653-3607
Karole.davidson@navy.mil
9

Dr. Daniel Wallace is a Senior Scientist at the Naval Surface Warfare Center,Dahlgren Division.
Dr. Wallace is the Human Factors Engineering Technical Warrant Holder for NAVSEA. He has
led development of human engineering methodologies that articulate processes and procedures to
be followed in executing standard human engineering practice within systems engineering. He is
active in executing the engineering rigor of this human engineering practice in a large number of
programs, as well as getting this message out to the engineering community through invited
briefings and in international forums. Internationally, he is the U. S. National Lead for a panel on
maritime human factors under The Technical Cooperation Program (TTCP) and is the chair of
the Human Systems Integration Working Group under the ABCANZ 99-02 panel.
Mr. Dennis White is the head of the Naval Surface Warfare Center Dahlgren’s Human Systems
Integration (HSI) branch, leading a team of Human Systems Engineers that address HSI in
numerous Navy, Joint and DoD system developments. The Dahlgren HSI team’sassets include
the Human Performance Laboratory (HPL) where systems and team concepts are developed and
evaluated for warfighter performance effectiveness in an operationally relevant environment. Mr.
White is the Manpower and Personnel Technical Warrant Holder for NAVSEA with extensive
experience in US Navy manpower modeling and simulation. He is a former Surface Warfare
Oficer and ‘Mustang’ –achieving a commission though the Navy Enlisted Commissioning
program.
Ms. Karole Davidson is the Human Systems Integration program director with Naval Surface
Warfare Center in Dahlgren, Virginia, where she has spent the last five years working on various
projects through the Naval Surface Warfare Center and other Navy customers. She currently
supports the Naval Sea Systems Command's Human Systems Engineering Directorate, NAVSEA
05H and various PEOs in the areas of policy, certification, and acquisition program support.
10