Computer Diagnostics - Siemens

Computer Diagnostics
Managing detailed health status information
of Healthcare’s Imaging Computer Systems
Imaging Computer Systems

Computer Diagnostics
is a crucial success
factor for delivering
and maintaining high
quality Imaging
Computer Systems
world-wide.
Introduction to CODIAG –
Computer Diagnostics
Siemens Healthcare’s modalities are in
use world-wide, 24 hours/day. It is important
for our customers, that downtimes
can be avoided or are at least planned
and that repairs can be easily achieved.
For Siemens itself, it is important to understand
the causes of failures and to be
able to consider these issues when designing
subsequent hardware generations.
Today, in many cases, hardware failures
cannot be proactively predicted and
complete systems are exchanged in the
field and sent back for repair. Sometimes,
when the repair centers analyzes returned
systems, they detect parts, which have
been exchanged in the field and have not
been qualified as spare parts or they detect
no hardware failure at all. These
systems could have stayed at the customer’s
site and most likely needed only a
reset to Siemens’ factory default settings
and/or a software re-installation.
Computer Diagnostics (CODIAG) was
developed by CV ME to address several
shortcomings in the design, manufacturing,
service and repair cycle. The main
focus of CODIAG is to synchronize the
way computer diagnostics is performed in
various locations, e.g. at factory shipment,
by a service technician in the field
or at the repair centers.
Conceptually CODIAG consists of software,
central databases and related services.
The software CODIAG is used to analyze
the current system status and compare
this report either with target values
or with reports from previous runs. The
databases store the configuration specification,
the factory shipment report and
the complete history of all reports from
each unit. The services provide additional
value, especially for Customer Service,
like proactive maintenance and remote
diagnostics support.
For full functionality, CODIAG needs the
systems to be connected to the common
Remote Service Platform and the installation
of vendor specific software to access
the hardware sensors, e.g. FTS Deskview.
Without these prerequisites, CODIAG is
still beneficial for Healthcare, but cannot
be used to its full potential.
Special Healthcare requirements
Due to the fact that Healthcare’s products
have to be fully FDA compliant, every
software that is shipped with our computers
must be validated and conform to
special requirements, different from typical
mainstream computer usage that
would be common in data centers.
CODIAG has been developed to conform
to the following mandatory requirements:
• no background tasks, must be passive if
not explicitly called from syngo software
or scheduled to run
• runs under all operating systems that
are used by Healthcare
• can be run unattended, without graphical
user interface (GUI)
• execution time shall be configurable
and tailored to use a very short time in
2

Important for the
customer perceived
quality of a modality is
not only the gantry,
but also a trouble-free
operation of the computer
systems operating
within.
order to be used frequently, without
much effort
• shall be utilized as service tool in the
field
• output must be structured and parseable
for easy post-processing analysis
• covers inventory, monitoring and
benchmarking functionality
• must be extensible to include Healthcare-specific
test functionality, e.g. receiver
boards or specific data loads
• shall include test functionality which
addresses common failures
Also, the diagnostics approach shall utilize
available commercial products and
resources and avoid duplicate implementation.
Off-the-shelf software analysis
A thorough survey and evaluation of
commercially off-the-shelf (COTS) software
has been conducted in 2007, before
deciding, which parts of the diagnostics
suite need to be developed in-house.
Key learnings from this survey were as
follows:
• most tools have their strength in one of
the fields (inventory, monitoring or
benchmarking), but shortcomings in
others
• no tool provided a structured, futureproof
output, that can be analysed
• no tool could perform a comparison to
a given requirement
• many monitoring tools needed to run
continuously in the background
• no tool provided an API to implement/extend
the functionality
• the best tools with a good mix of functionality
used DOS as OS and were not
useable during normal operation
The survey results were discussed within
Healthcare and it was decided to do an inhouse
development of the core logic and
use vendor tools, open-source algorithms
and existing Healthcare tools to achieve
full functionality. Since the majority of
Siemens’ installed base of imaging computers
consists of FTS systems, using
Deskview to retrieve hardware sensor
data was a logical choice. FTS has been
contracted to modify Deskview to run in
passive mode and only consume calculation
power, when explicitly called. For
other platforms and operating systems,
different methods are used to retrieve
hardware information, e.g. IPMI or direct
sensor register calls.
3

Computer diagnostics
test should work their
way up the Diagnostics
Pyramid: quick tests
first, then investigate
in more detail,
if necessary.
System design
Root causes for computer system failures
can sometimes be easily detected or require
extensive benchmarking and analysis
in other cases. In order to most efficiently
narrow down a symptom to the
root cause, a systematic approach has
been developed, that operates on several
levels, which are shown in the Diagnostics
Pyramid below.
An analysis showed, that many problems
can be detected, just by taking an inventory
snapshot of the system and compare
the found components, devices and setting
with a given specification. This scan
takes only a couple of seconds and can
already reveal a lot of information, when
compared with previous runs. The Inventory
level basically answers the question:
„Is everybody there?“
Tests on the Monitoring level take longer,
but reveal information about the current
health status and changing dynamic data.
Examples are: CPU temperature, fan
speeds, number of remapped disk sectors,
SMART trips, network errors etc.
These are answers to the question: „Is
everybody doing well?“
The Diagnostics level tests the hardware
according to load profiles, tailored to the
particular usage scenario of that system.
These tests will typically not be run every
day, but can be useful during a scheduled
service period. These tests can easily run
Order of Test Execution
Duration/Cost of Tests
several minutes to several hours. They
can cover CPU stress tests, hard disk performance
tests, memory or network tests
and especially combinations of these
tests. The question to be answered here
is: „Is everybody performing okay?“
Eventually, CODIAG could evolve into an
Expert System in the future. Currently,
planned features for the next version is to
fully integrate CHSBench which is a standalone
tool today.
Internal Architecture
From the start, the core of CODIAG was
developed to be as operating system and
platform independent as possible. Analyzing
the defects found in our repair center,
we defined a set of items that should
be inventoried and monitored. The underlying
data source for each item varies
between hardware platforms and operating
systems.
Currently we use three data providers:
• operating system’s standard information
sources, e.g. WMI, process tables,
registry etc.
• vendor tools, e.g. FTS Deskview
• system category explorer (SCE), a library
interface to access hardware information,
which is normally not exposed
via the other two methods, e.g.
hard disk settings, cache settings etc.
Expert
System
System Analysis
»What is wrong?«
»How can it be fixed?«
Diagnosis
Performability Check
»Is everybody fit & performing OK?«
Monitoring
(of „technical“, dynamic parameters)
Availability Check
»Is everybody doing well?«
Inventory
(of „logistical“, static parameters)
Visibility Check
»Is everybody there?«
Expected Frequency of Faults
Future
FY
10/11
FY
09/10
FY
08/09
CODIAG
&
CHSBENCH
CODIAG
(combined with
FTS DeskView
and other
vendor tools)
Timeline Tools
4

OS
DiagInv
(Inventory, Monitoring)
Vendor Tools
System Category
Explorer
Imaging Computer System Hardware
The information provided is then used by
the DiagInv Module to create the report
and do the comparison with expected
values or values from previous runs. Depending
on the settings in the configuration
specification, DiagInv can also raise
errors or warnings, when a deviation is
found.
All configurations and reports are stored
in structured eXtensible Markup Language
(XML) files which conform to a predefined,
versioned XML schema. It is very
straightforward to parse these files for
further analysis and processing.
External Architecture
CHSBench
(Diagnostics, Performance)
Typically a Healthcare business unit decides
together with CV ME on the set of
system characteristics to be evaluated
and reported. The interval and particular
points in time when CODIAG shall be run
are also defined together with the BU to
ensure minimal interference with the
normal modality operation. For the start
CV ME recommends to implement CODI-
AG runs at the start and/or shutdown of
the modality, which would – in most
cases – result in daily runs.
OS
Once the reports have been generated,
the syngo Autoreport functionality will be
used to pick up and transfer the files to
Architecture model of
Computer Diagnostics.
DiagInv implements
the core logic, comparison
functionality and
reporting. Additionally
inventory and monitoring
are collected.
CHSBench can also be
used independently for
performance benchmarking.
Siemens. Individual subscription servers
within the Siemens Healthcare network
will then receive these files. CV ME will
host a database where these reports are
consolidated and stored for further processing.
If system management (hp OpenView
and the common Remote Service Platform)
is available at a site, additional
functionality can be deployed. CODIAG
can be configured to raise an error or
warning on certain conditions, e.g. a fan
failure or overheating/throttling of the
CPU. Based on system management
templates these warnings can be detected
and so-called events can be generated.
These events are transmitted in real-time
to Healthcare and the existing processes
of the Uptime Service Center can be used
to start an investigation, inform the
customers or initiate a Remote Diagnostics
session to confirm or correct
the error or schedule a site visit of a service
technician.
5

Customer
Requirement
Specification
1
4
1.4.2010
2.4.2010
Field
3.4.2010 Snapshot
Field
Snapshot
Field
Snapshot
1.9.2013
Engineering
Field
Snapshot
Configuration
Specification
2
Hospital
Repair
&
Manufacturing
Factory
Reference
Snapshot
Recycling
Factory
Reference
Snapshot
Factory
Reference
Snapshot
Operations concept and services
Basis for the operations concept is the
uniform and continuous collection of
system diagnostics data. This data is the
foundation of all further activities and
services.
Basic operations concept
3
During the first two phases (engineering
and manufacturing phases) CODIAG is set
up in general for each class of system and
subsequently for each individual system.
① In the initial project phase CV ME receives
the customer’s requirement
specification and starts the system selection
and validation project
② Hardware engineers create a formal,
structured representation of the requirements
in XML format as „Configuration
Specification“. This specification
contains e.g. the required capacity or
throughput requirement of the storage
subsystem.
③ During manufacturing, in the final
software installation and test phase
the assembled system configuration is
taken as „Factory Reference Snapshot“.
This is the most detailed snapshot,
5
1.9.2013
Engineering
Field
Snapshot
Configuration
Specification
Hospital
Repair
&
3.9.2013
Repair
Snapshot
Manufacturing
Factory
Snapshot
containing also serial numbers of the
system.
The third phase is the continuous daily
operation in the hospital and further
phases deal with unexpected events.
④ In the hospital at the configured points
in time, a snapshot is taken, similar to
the Factory Reference Snapshot. Depending
on the business unit, a comparison
with previous runs and/or with
the Configuation Specification or Factory
Reference Snapshot can be done,
but this is not mandatory. In any case,
the resulting snapshot is prepared for
transmission to the Siemens Headquarter
and syngo Autoreport mechanisms
take care of the transmission.
⑤ If an error occurs in the field and the
system cannot be repaired locally, it is
sent back to CV ME’s repair centers.
The last Field Snapshot may be still on
the system or already transferred to
the HQ.
⑥ After arrival, a repair snapshot is taken.
This snapshot can now be compared to
the Configuration Specification, the
Factory Reference Snapshot and the
last Field Snapshot. These comparisons
now enable a more detailed investiga-
7
Snapshot
Deletion
Recycling
6
6

Computer diagnostics
test should work their
way up the Diagnostics
Pyramid: quick tests
first, then investigate
in more detail,
if necessary.
tion, what happened to the system,
than previously possible. After repair,
the system can again be deployed in
the field.
⑦ Eventually, if the system cannot be
repaired, it will be recycled. All
snapshots of this particular system will
be removed from the databases, but of
course the base Configuration Specification
and snapshots of other systems
still in the field are unaffected.
Proactive Maintenance
With the availability of consistent diagnostics
data, further services can be
offered by CV ME. The statistical analysis
and trend analysis of data will be used to
create predictions of components where a
failure might be imminent. Proactively CS
can be informed, that a service site visit
should be scheduled or a replacement
part could be dispatched to the responsible
service technician or even the end
customer, if so desired.
Remote Diagnostics
The latest hardware generations provide
technologies that enable to remotely
operate and diagnose a malfunctioning
computer system. Of course this depends
on the criticality of the error, e.g. if the
power supply is broken, remote diagnostics
cannot be used either. But in
many cases skilled technicians can remotely
reboot the system, run tests and
perform error correction measures.
Reduction of
overall system
failures
A probability analysis showed, that in
48% of all computer failures – if remotely
analyzed – a complete system exchange
could possibly be avoided.
The usage of Remote Diagnostics depends
on a certain infrastructure on
Healthcare’s and the customer’s site, so
the deployment of this service is expected
to more slowly ramp up over time, than
the other services, Proactive Maintenance
and Guided Local Diagnostics. Still, this
service delivers the greatest benefits to
CS and the customer.
Guided Local Diagnostics
Since the length of an on-site visit of a
service technician is one of the largest
cost drivers, CODIAG was designed, to
performa a broad, but shallow depth
analysis first in a very short time to detect
a deviation from an expected
configuration very efficiently. If combined
with Remote Diagnostics or Proactive
Maintenance to already identify a failed
part time, the service technician can
replace the spare part and then quickly
test the repaired system, and check
whether the configuration is valid.
Proactive
Maintenance
Continuous system monitoring, transfer of data
to Siemens HQ for analysis and predictions
Increase part
exchange,
reduce system
swap
Guided Local
Diagnostics
Local analysis tools, tailored to Siemens systems and
boards, requirement check and factory comparison
Reduction of
field service
dispatch
Reduction of
on-site
maintenance
time
Remote
Diagnostics
Remote network connect to system, diagnostics,
repair or spare part dispatch, inform technician
7

www.siemens.com