Medical Visualization with ITK - Scientific Computing and Imaging ...

Medical Visualization
Introduction to Segmentation with ITK
Josh Cates
with the Insight Consortium

Goals
Supplement Software Guide with theory, ideas,
and examples.
Assembling applications from ITK components
• GUIs
• Visualization
Scientific Computing and Imaging Institute, University of Utah

Medical Image Segmentation
Partitioning images into meaningful pieces,
e.g. delineating regions of anatomical
interest.
Huge area of research
Some common approaches
• Edge based – find boundaries between
regions
• Region based – similarity of pixels within a
segment
• Pixel classification – metrics classify regions
(e.g. tissue types)
Scientific Computing and Imaging Institute, University of Utah

Seg. in ITK: An Image Processing Pipeline
Preprocessing
Raw Data
Filtering
linear
nonlinear
Feature
Extraction
differential geom.
edge detection
ITK segmentation filters are not
complete applications –
components in a pipeline.
How ITK filters fit together
to produce segmentation pipelines.
Segmentation
Visualization
region growing
binary volume
watersheds
meshes
level-sets
labeled image
implicit surfaces
Scientific Computing and Imaging Institute, University of Utah

Seg. in ITK: An Image Processing Pipeline
Raw Data
Filtering
User
Interface
Feature
Extraction
Segmentation
Visualization
Scientific Computing and Imaging Institute, University of Utah

Features of ITK Filters
Most filters are N dimensional
• Image = volume
• Pixel = voxel
Data type matters – cast if necessary
Watch spacing – resample if necessary
Read filter's documentation
● And send feedback!
Scientific Computing and Imaging Institute, University of Utah

Preprocessing for Segmentation
Importance of Preprocessing
• Data filtering is part of segmentation
­ Cropping
­ Resampling
­ Seed points, regions
• Filtering introduces “hidden parameters”
­ Requires expertise
­ Tuned for application
Scientific Computing and Imaging Institute, University of Utah

Preprocessing for Segmentation
Goals of Preprocessing
• Prepare data for segmentation
• Remove noise & uninteresting detail –
downsample / blur
• Target interesting information – crop, seed
points
• Preserve and extract features – edges
Scientific Computing and Imaging Institute, University of Utah

Linear vs. Nonlinear Diffusion
Gaussian blurring
• Convolution framework
• Recursive Gaussian
• Accuracy / Speed tradeoffs
Nonlinear diffusion
• PDE framework
• Anisotropic, curvature­limited, bilateral
● P. Perona, J. Malik, Scale-space and edge detection using anisotropic diffusion,
IEEE Transactions on Pattern Analysis Machine Intelligence 12 (1990) 629–639.
● R. Whitaker, X. Xue. Variable-Conductance, Level-Set Curvature for Image
Processing. International Conference on Image Processing. IEEE, 2001.
● Tomasi, Manduchi. Bilateral Filtering for Gray and Color Images. IEEE ICCV. 1998.
Scientific Computing and Imaging Institute, University of Utah

Linear Diffusion
Destroys and moves edges
Scientific Computing and Imaging Institute, University of Utah

Nonlinear Diffusion
Preserves Edges
Scientific Computing and Imaging Institute, University of Utah

Understanding and Using ITK Segment. Filters
✔
Region Growing
✔
Morphological Watersheds
✔
✔
Level Set Methods
Combination Methods – extend seg. pipeline
Other algorithms available!
Scientific Computing and Imaging Institute, University of Utah

Segmentation Applications – Ideas & Examples
✔
Interactive Level Sets with Fltk
✔
User­assisted Watersheds with Tcl and VTK
✔
Validation Work
Scientific Computing and Imaging Institute, University of Utah

Region Growing
Idea
• Start with set of seed pixels – region
• Iteratively include neighboring pixels that
satisfy membership criteria
Membership criteria – similarity based metrics
• Intensity interval
• Regional statistics
Flood­fill algorithms
Scientific Computing and Imaging Institute, University of Utah

Region Growing Filters in ITK
http://www.itk.org/itkSoftwareGuide.pdf
Connected Threshold
• Include pixels inside intensity interval
Neighborhood Connected
• More conservative – includes pixels only if
all pixel neighbors are inside an interval
Fuzzy Connectedness
• Statistical method assigns affinities between
pixels
• Combines classification & region approach
● J. Udupa, S. Samarasekera. “Fuzzy connectedness and object definition: Theory,
algorithms, and applications in image segmentation.” Graphical Models in Image
Processing, 58(3): 246-261, 1996
Scientific Computing and Imaging Institute, University of Utah

Confidence Connected Filter
Threshold based region growing
Mean and standard deviation of region
determine upper and lower thresholds
Recomputes thresholds at intervals
Compute m and s
of region
Flood fill with threshold interval
[m-ks, m+ks]
Repeat N times
Scientific Computing and Imaging Institute, University of Utah

Example: Confidence Connected Filter
Examples/Segmentation/ConfidenceConnected.cxx
input file
name
iterations
time step
iterations
multiplier
seed point
output file
name
Image
Reader
Curvature
Flow
(smoothing)
Confidence
Connected
Image
Writer
Scalar
Image
Binary
Image
Scientific Computing and Imaging Institute, University of Utah

Example: Confidence Connected Filter
Examples/Data/BrainProtonDensitySlice.png
original
white matter
(60,116)
ventricle
(81,112)
gray matter
(107,69)
smoothing iterations 5
smoothing time step 0.125
C.C. multiplier 2.5
C.C. iterations 5
Scientific Computing and Imaging Institute, University of Utah

Morphological Watersheds
Large body of research over 20 years
Inspired by hydrology – treat image as
landscape and look for watersheds
Watershed Transform – alg. that identifies
watershed regions
Main Variations
✔ Top­down: classify pixels by shortest
topological distance to local minima
✗
Bottom­up: simulated immersion algorithms
● L. Vincent, P. Soille,Watersheds in digital spaces: An efficient algorithm based on
immersion simulations, PAMI 13 (6) (1991) 583–598.
Scientific Computing and Imaging Institute, University of Utah

ITK Watershed Transform
Image treated as a topological relief map –
intensity represents height
Gradient descent defines segmented regions
• Set of all pixels whose paths of steepest
descent terminate in same local minimum
• Bounded by image features
Global – operates on entire image at once
“No parameters”
Scientific Computing and Imaging Institute, University of Utah

The Watershed Transform
Image (filtered)
Feature Extraction
“Edge Map”
Watershed Transform
Watershed Depth
Scientific Computing and Imaging Institute, University of Utah

The Oversegmentation Problem
Watershed transform produces too many
regions
• One per local minimum
• Especially in noisy or highly detailed data
To alleviate oversegmentation
✗
✔
Immersion alg. from marker points
Hierarchical approach – merge adjacent
regions according to increasing watershed
depth
● A. P. Mangan, R. T. Whitaker, Partitioning 3D surface meshes using watershed
segmentation, IEEE Transactions on Visualization and Computer Graphics 5 (4)
(1999) 308–321.
Scientific Computing and Imaging Institute, University of Utah

Watersheds Hierarchy
Enforce minimum watershed depths
at successively higher levels.
Watershed Depth Threshold
Oversegmented Undersegmented
Watershed Transform
= basin
Boolean Operations
On Sub-trees
(e.g. user interaction)
Initial Watershed
Transform
Watershed Depth
Scientific Computing and Imaging Institute, University of Utah

Hierarchy Level 1
Scientific Computing and Imaging Institute, University of Utah

Hierarchy Level 2
Scientific Computing and Imaging Institute, University of Utah

itk::WatershedImageFilter
Watershed Image Filter
Height
image
Segmenter
(watershed
transform)
Catchment
Basins
Image
relabeler
Labeled
image
Lower Thresholding
Process object
Data object
Parameter
Hierarchy
generator
Hierarchy
Maximum Watershed Depth
Output Depth (Level)
Scientific Computing and Imaging Institute, University of Utah

Example: Morphological Watersheds
Examples/Segmentation/WatershedSegmentation1.cxx
file
name
iterations
conductance
time step
level
threshold
file
name
Image
Reader
Vector
Cast
Anisotropic
Diffusion
Gradient
Magnitude
WS
Image
Filter
“Colorizer”
Image
Writer
RGB
Image
labeled
Image
(u. long)
RGB
Image
Scientific Computing and Imaging Institute, University of Utah

Watershed Filter Output
Output level recomputes real­time
Format
• Labeled N dimensional image
• Unsigned long integers
• Potentially large number of values!
Visualization?
• Contour lines
• Map to RGB
Scientific Computing and Imaging Institute, University of Utah

Mapping Labels to RGB for Visualization
Scientific Computing and Imaging Institute, University of Utah

Constructing a “Colorizing” Filter
Input
Image
(unsigned
long)
Unary Functor
Image Filter
ScalarToRGB
Functor
Output
Image
(RGB)
Scientific Computing and Imaging Institute, University of Utah

Setting Watersheds Parameters
original image
Diffusion
Iterations
Watershed Depth (Level)
10
15
0.01
0.02 0.03 0.04 0.05 0.06
20
conductance 1.0
lower threshold 0.0
30
Scientific Computing and Imaging Institute, University of Utah

Setting Watersheds Parameters
Watershed Depth (Level)
original
image
0.01
0.02
0.03
conductance 1.0
diffusion iterations 10
lower threshold 0.0
0.04
0.05
0.06
Scientific Computing and Imaging Institute, University of Utah

Setting Watershed Parameters
Lower Threshold
0.00001 0.00010 0.00100 0.01000 0.10000
original
image
Diffusion Conductance
0.25 0.50 1.00 1.25 1.50 2.00
diffusion iterations 10
level 0.03
Scientific Computing and Imaging Institute, University of Utah

Generating Models
Extract Segmented Regions
• Thresholding, connected components
• Produce binary volumes
Visualization
• Overlays
• Surface Rendering
Scientific Computing and Imaging Institute, University of Utah

Watersheds GUI Design
InsightApplications/SegmentationEditor
User
Tk Graphical User Interface
Tcl Wrapper
vtkITK IP Pipeline
VTK Vis. Pipeline
}
Input
Data
Scientific Computing and Imaging Institute, University of Utah

Demo: Watersheds GUI
Watershed
transform
Data with
overlay
Segmentation
in progress
Sliders manipulate
watershed depth
and position in the
hierarchy.
3D isosurface
rendering
Watershed Depth Threshold
Scientific Computing and Imaging Institute, University of Utah

vtkITK
InsightApplications/vtkITK
Mechanism for converting ITK filters into VTK
filters
VTK wrapped for Python, Tcl, Java
VTK
Image
VTK Image Import
vtkITKImageToImageFilter
ITK
Image
ITK Filter
ITK
Image
VTK Image Export
VTK
Image
Scientific Computing and Imaging Institute, University of Utah

Validation: User Study
Comparison of user-assisted hierarchical
watersheds with hand-contouring
Hand contouring
• De facto standard
• General and reliable(?)
Issues
• Can a general purpose segmentation
algorithm compete?
• (Are our validation tools up to the task?)
Cates, Whitaker, Jones, “Case Study: An Evaluation Of User­Assisted Hierarchical
Watershed Segmentation”, Medical Image Analysis, Under review.
Scientific Computing and Imaging Institute, University of Utah

User Study Overview
MRI Brain Tumor
(4 cases)
Ground Truth
Subjects (Slicer)
Slice from HBW BT
database (4 per
case)
WS Segmentation
Subjects
Radiologists (3) from
Univ of Utah
VHP Cryosection
(Eyeball, optic nerve,
lateral rectus)
3rd­year med.
students at HBW
and Utah (EB–4,
ON–3, LR–8)
3rd­year med.
students at Utah (7)
Scientific Computing and Imaging Institute, University of Utah

Validation Strategy
Experts
Hand Contouring
STAPLE
Ground
Truth
User Studies
WS
(RR vs Full)
Benchmark
Performance
Parameters
Accuracy
Precision
Efficiency
WS
Performance
Parameters
Analysis
Conclusions
Scientific Computing and Imaging Institute, University of Utah

Validation Results
Eyeball
Optic nerves
Data Hand contour Watersheds
Scientific Computing and Imaging Institute, University of Utah

Validation Results
Hand contour
Watersheds
Scientific Computing and Imaging Institute, University of Utah

Validation Results
Brain tumor MRI
Visible Human cryosections
Scientific Computing and Imaging Institute, University of Utah

Summary of Validation Results
Accuracy
• Sensitivity (TPF) is generally low
• Total correct fraction generally within variation of
experts (better for tumor data)
• Generally better than level-set approach
Precision
• Significantly better than both hand contouring and
level­set
Efficiency
• Versus hand contouring, no comparison (30 min
vs. 2­3 hours)
• Versus level­set, more preprocessing and
comparable user times
• Time/expertise to tune hidden parameters issue
Scientific Computing and Imaging Institute, University of Utah

Validation Conclusions
Watershed Segmentation
• WS probably makes more sense vs hand
contouring in many applications
• True positive fraction is an issue – could
tune for that metric
Validation
• Rich set of systematic tools
• Pixel­based – shape metrics lacking
• Hand contouring for ground truth
questionable
Scientific Computing and Imaging Institute, University of Utah

Level­Set Surface Modeling Theory
k th Level Set: set of all points of value k
Embed N dimensional surface as Zero level set
of N+1 dim. volume
Model N dim. surface movement as an evolving
wavefront – forward differences solution to
PDE
t
=−F∣∇ ∣
Scientific Computing and Imaging Institute, University of Utah

Segmentation Using Level Sets
Define speed term(s) to go to zero at edges –
data fitting term
Surface motion/speed based on image features
or intensity
Solve the level­set equation where
F =Speed −Curvature
Scientific Computing and Imaging Institute, University of Utah

Insight PDE Solver Framework
Purpose
• Nonlinear image processing – e.g.
anisotropic diffusion
• Moving wave fronts – level set models
• Deformable registration
Generic framework
• Separate solvers from equations –
interchangeable code objects
• Focus on infrastructure & include instances /
examples in the toolkit
Scientific Computing and Imaging Institute, University of Utah

PDE Solver Hierarchy
Finite Difference
Solver
Finite Difference
Function
Dense
Narrow
Band
Sparse
Threaded
Sparse
Diffusion
Level
Set
Segment.
Diffusion
Deform.
Registration
Other Solvers
Aniso. Diff Curv.
Limited
Other Functions
4 th Order
Scientific Computing and Imaging Institute, University of Utah

Constructing a PDE Filter
Subclass solver for application – parameters,
convergence criteria
Virtual method plugs in function object
Typically only need a header file
Input
Image
Solver Object Subclass
Function
Object
Output
Image
(Filtered)
Parameters
Scientific Computing and Imaging Institute, University of Utah

Anisotropic Diffusion Framework
Finite Difference Solver
Finite Difference Solver
Dense Solver
Anisotropic Diffusion Filter
User-Defined Diffusion Filter
Diffusion
Function
Input
Image
Perona-
Malik
Function
Curvature-
Limited
Function
Vector-
Valued
Function
Output
Image
Scientific Computing and Imaging Institute, University of Utah

Level­Set Segmentation Framework
Sparse or Narrowband solver
Base LS function
• Generic equation
• Virtual Speed Term methods
Combine different Speed terms in plug and play
manner
Speed:
Data Fitting
Term
Regularization/
Smoothing
Surface
Motion/
Segmentation
Scientific Computing and Imaging Institute, University of Utah

Level­Set Segmentation Framework
Finite Difference Solver
Finite Difference Solver
Sparse-Field Level-Set Solver
“Feature”
Image
Level-Set Segmentation Filter
Curvature
Function
User-Defined LS Seg. Filter
Level Set
Function
Output
Model
Initial
Model
Shape
Detection
Function
Canny
Edges
Function
Active-
Contours
Function
Laplacian
Function
Threshold
Function
Scientific Computing and Imaging Institute, University of Utah

Example: Threshold­based LS Segmentation
Speed function (positive inside object)
Similar to confidence connected filter
Points Outside
Points Inside
Points Outside
Model Speed
Image Intensity
Low
Threshold
High
Threshold
•Lefohn, Kniss, Hansen, Whitaker, “GPU­Based, Interactive, Level­Set Models for Volume
Visualization and Analysis”, IEEE Vis 2003
•Lefohn, Whitaker, Cates, “Interactive Level­Set Models for Brain Tumor Segmentation”,
MICCAI 2003
Scientific Computing and Imaging Institute, University of Utah

Example: Level­Set Segmentation
Examples/Segmentation/ThresholdSegmentationLevelSetImageFilter.cxx
Fast
Marching
OR
Image
Reader
seed
indicies
Image
Reader
distance to
surface from
seeds
LS
Segment.
Filter
file
name
Image
Writer
LS
Image
(float)
Initial
model
Feature
Image
number of iterations
maximum rms change
curvature scale
feature scale
Scientific Computing and Imaging Institute, University of Utah

Level­Set Filter Output
Format
• Floating point
• Zero isosurface
t
• Inside negative, outside positive
Visualize?
• Render zero isosurface
• Zero crossing – color map, overlays
Scientific Computing and Imaging Institute, University of Utah

Example:Isosurface Overlays in VTK
InsightApplications/LevelSetSegmentation/vtkViewOutput.tcl
Speed
Image
Image
Reader
Resampler
Image
Mapper
LS
Image
Image
Reader
Threshold
Filter
Resampler
Image
Mapper
2D
Actor
Viewer
Original
Image
Image
Reader
Resampler
Scientific Computing and Imaging Institute, University of Utah

Example: Level­Set Segmentation Results
Original
White matter Ventricle Gray matter
(60,116)
150
180
(81,112)
210
250
(107,69)
180
210
: Seed Point
: Lower thresh.
: Upper thresh.
Scientific Computing and Imaging Institute, University of Utah

Interactive Level Sets
InsightApplications/ThresholdSegmentationLevelSetFltkGUI
User
ITK IP Pipeline
Fltk Graphical User Interface
OpenGL Vis.
}
Input
Data
Scientific Computing and Imaging Institute, University of Utah

Fast Light Toolkit (Fltk)
Cross­platform, open­source C++ GUI toolkit
● Windows, Linux / Unix (X11), MacOS X
GNU Library General Public License (LGPL)
(more restrictive than ITK license)
Includes “Fluid” UI builder application
ITK image viewer widgets
● InsightApplications/Auxiliary/FltkImageViewer
VTK – Fltk interface code
● http://vtkfltk.sourceforge.net/
http://www.fltk.org
Scientific Computing and Imaging Institute, University of Utah

Threshold LS Segmentation Demo
Interface to segmentation pipeline
Setting parameters
Effects of curvature
Visualization
Scientific Computing and Imaging Institute, University of Utah

Event Handling in ITK
Command / Observer design pattern – generic
callback mechanism
Commands are registered as observers of a
filter's events
Filters invoke events, triggering command
execution
User Code
MyCommand
Execute()
AddObserver(IterationEvent(), MyCommand)
ITK Filter
...
InvokeEvent(IterationEvent())
InvokeEvent(ProgressEvent())
...
Scientific Computing and Imaging Institute, University of Utah

How To Visualize Evolving Level­Set Surfaces
See: InsightApplications/ThresholdSegmentationLevelSetFltkGUI/SegmenterConsole.cxx
Write an ITK Command object to render LS filter
output
● Be sure to detach output from pipeline, use SetPixelContainer
Register your command as observer of LS
filter's IterationEvent()
myRenderCommand
itk::image::Pointer output
output -> SetPixelContainer(myFilter->GetOutput())
Render(output)
GUI Code
myFilter->AddObserver(IterationEvent(), myRenderCommand)
Level-Set Filter
“myFilter”
...
InvokeEvent(IterationEvent())
...
Scientific Computing and Imaging Institute, University of Utah

Example: Multiscale 3D Segmentation
Scale
Seed surface
Data
1/4
1/2
1/1
Scientific Computing and Imaging Institute, University of Utah

Advanced Features in the PDE Framework
Parallel Solvers – Narrowband, Sparse field
Speedup
(vs. 1 processor
50
45
40
35
30
25
20
15
10
5
0
SGI Origin 3000
64 600 Mhz Processors
0 20 40 60
Number of processors
Scientific Computing and Imaging Institute, University of Utah

Surface Processing with 4 th Order PDEs
Isotropic
Anisotropic
•T. Tasdizen, R. Whitaker, P. Burchard, S. Osher, “Geometric Surface Smoothing via
Anisotropic Diffusion of Normals”, Proc. IEEE Visualization 2002, pp. 125–132.
•T. Tasdizen, R. Whitaker, P. Burchard, S. Osher, “Geometric surface processing via normal
maps”, To appear: ACM Trans. on Graphics.
•T. Tasdizen, R. Whitaker, “Higher-Order Geometric Surface Priors for Surface Estimation”,
IEEE Trans. on Pattern Analysis and Machine Intelligence, To appear.
Scientific Computing and Imaging Institute, University of Utah

4 th Order Flow Segmentation Framework
Sparse-Field Level-Set Solver
(Refitting)
Finite Difference Solver
Finite Difference Solver
Manifold Solver
(Normals Processing)
Input
Image
Level-Set Segmentation Filter
Curvature
Function
User-Defined LS Seg. Filter
Speed
Function
Initial
Model
Shape
Detection
Function
Canny
Edges
Function
Active-
Contours
Function
LaPlacian
Function
Output
Model
Threshold
Function
Scientific Computing and Imaging Institute, University of Utah

Segmentation Using 4 th Order Flows
Speed term only
Speed + Anisotropic
4 th order terms
Scientific Computing and Imaging Institute, University of Utah

Recent Development – CVS
LS Solvers interchangeable at run­time
• Choose between sparse­field, parallel, and
4 th order
Stop / Start capability for solvers
• Optionally maintains state between updates
Better access to speed function
• Supports user interface development
• Optionally precalculate
• Plug user speed function into base filter
Scientific Computing and Imaging Institute, University of Utah

Example: Combining Segmentation Filters
Generate initial model using confidence
connected filter
Fit to data using level­set methods – minimize
distance to Canny edges
Confidence
Connected
Initial
model
Feature
Image
Canny LS
Segmentation
Filter
LS
Image
(float)
Image
Anisotropic
Diffusion
Scientific Computing and Imaging Institute, University of Utah

Harvard Brigham & Womens Tumor Database
HBW Surgical Planning Lab
T1 MRI brain tumor patients
from clinical database.
Includes meningioma, low-grade
glioma, astrocytoma.
Case 1: Left-frontal meningioma
Warfield, Nabavi, Butz, Tuncali, Silverman, “Intraoperative segmentation and nonrigid
registration for image guided therapy, in: MICCAI'2000, SpringerVerlag, 2000, pp.176­
185.
Scientific Computing and Imaging Institute, University of Utah

Confidence Connected + Level­Set Result
Initial confidenceconnected
result
LS Speed Term: distance
from Canny edges
Post-processing with
Canny LS segmenter
Scientific Computing and Imaging Institute, University of Utah

Acknowledgements
Ross Whitaker, Tolga Tazdizen, Darby Van Uitert,
Suyash Awate, Xinwei Xue: Utah ITK Team.
Drs Simon Warfield, Michael Kaus, Ron Kikinis, Peter
Black, and Ferenc Jolesz: Brain tumor database.
Greg Jones & Drs Steve Stevens, Troy Marlow, Jay
Tsuruda, Peter Ratiu: Validation Study.
National Library of Medicine
Insight Users
Scientific Computing and Imaging Institute, University of Utah

More Information...
enjoy ITK!
http://www.itk.org
http://www.sci.utah.edu/~cates
Scientific Computing and Imaging Institute, University of Utah