X - Vision at IME-USP

Design of Morphological
Operators by Learning
Junior Barrera
jb@ime.usp.br
IME-USP

Layout
Introduction
Binary operator design: W-operators W
Binary operator design: constraint W-W
operators
Gray-scale operator design: apertures
Gray-scale operator design: stack filters
Conclusion
2

Introduction

Morphological Machine (MMach)
d e A A n
i Ÿ ne
Ÿ ⁄ ∏
A
4

Properties
Any finite lattice operator can be
implemented as a program of a MMach
Finite lattices of practical importance are the
lattice of binary and gray-scale images
5

Morphological Toolbox
Library of hierarchical functions:
- primitives: : elementary operators and
operations;
- high order operators: : primitives and high
order operators
6

Heuristic Design
Divide the problem in subproblems
Each subproblem is solved by a toolbox
function
Integrate the subproblems solution
7

Automatic Design
Operator learning in a standard
representation
Finding an equivalent and more efficient
representation
8

Operator Design
9

Application
10

Change of Representation
e ⁄ e
⁄ Ä Ä Ä ⁄ e
A 1 A 2 A n
y
Operators
d A e A
Phrases
11

Operator Design

The problem
observed
ideal
Find an image operator that transforms the observed
image to the respective ideal (or “close to the ideal”)
image.
13

Binary image operators
Binary image :
Binary images can be understood as sets :
is a complete Boolean lattice
Binary image operators = set operators :
14

Translation invariance
Translation of S by z :
is
translation-invariant iff
15

Local definition
Window :
An image operator is locally defined within iff
16

W-operators
Translation invariance
+
local definition within
=
-operators
z
W-operators are characterized by Boolean functions.
17

Window
Representation
1
1
1
1
X11 1X0 11X
18

Statistical Hypothesis
X and Y are jointly stationary
P( S ∩Wz,
Y
)
is the same for any z in E
19

Stationary Process
20

Join Stationary Process
21

Error measure
Design goal is to find a function
with minimum risk.
Risk (expected loss) of a function :
Loss function
X is a random set
Y is a binary random variable
22

MAE example
Example : MAE loss function
Optimal MAE function
ψ ( X
)
=
⎧
⎨
⎩
1
0
p (1,
p (1,
X )
X )
>
≤
p (0,
p (0,
X )
X )
23

Design procedure
24

PAC learning
L is Probably Approximately Correct (PAC)
For m > m( ε,
δ ) examples
Pr(|
R(
ψ
)
− R(
ψ ) | < ε)
> 1
opt
−
δ
ε,
δ ∈ (0,1)
25

Edge detection
Training images
Test images
26

Noise filtering
Training images
Test images
27

Texture extraction (1)
Training images
28

Texture extraction (2)
Test images
29

Fracture Detection
Training images
Test images
30

Amount of data available
18,10 %
3,13 %
2,68 %
2,21 %
Last Image
2,01 %
1,82 %
1,69 % 1,60 %
Noise 1 image 2 images 3 images 8 imagess 13 images 25 images 32 images
31

window 5x5, 6 training images
addition: 2%
subtraction: 1%
distict patterns : 140.060
in 1.548.384
addition: 3%
subtraction: 3%
distinct patterns : 266.743
em 1.548.384
addition: 6%
subtraction: 6%
distinct patterns: 487.494
in 1.548.384
32

Size of the window
Error rate in function of window size
0.70%
0.60%
0.50%
0.40%
0.30%
0.20%
0.10%
0.00%
3x3 4x4 5x5 6x6 7x7
1
2
3
4
5
6
7
8
33

Constraint Design

Constraints
ψ opt
Ψ
35

Constraints
ε des
ε des-con
Error
ε opt-con
ε opt
N 1 N 0
N 2
Sample size
36

Constraints
Structural Constraints
- impose maximum number of elements in the basis
- use alternative structural representations
(e.g., sequential)
37

Constraints
Algebraic constraints
- consider a class of operators satisfying a given
algebraic property (e.g., increasingness ,
idempotence, auto-dualism, etc)
- consider a constraint by a multi-resolution criteria
- consider a constraint by an envelope of operators
- consider a constraint by a multi-resolution envelope
criteria
38

Structural Constraint

Minimum number of intervals
in the basis

Iterative design

Motivation : composition of operators over small
windows produces an operator over a larger window
is a
-operator
45

Application example
test image
iteration 1
iteration 2
46

Application example
test image iteration 1
iteration 2 iteration 3
iteration 4 iteration 5 iteration 6
47

Algebraic Constraint

Increasing W-operators
increasing
non-increasing
49

Design of increasing W-operators
increasing
non-increasing
increasing
50

Multi-resolution constraint

x 1 x 2 x 3 x 4 x 5 x 6 x 7 x 8
8 variables
z 1 z 2 z 3 z 4
4 variables
w 1 w 2 2 variables
0
1
0 1
00
01
10
11
0
0
0
1
1
0
1
1
0000
0001
0010
0011
0100
0101
0110
0111
1000
1001
1010
1011
1100
1101
1110
1111
0
0
0
0
0
0
0
1
0
0
1
0
0
0
1
1
0
1
0
0
0
1
0
1
0
1
1
0
0
1
1
1
1
0
0
0
1
0
0
1
1
0
1
0
1
0
1
1
1
1
0
0
1
1
0
1
1
1
1
0
1
1
1
1
2 variables: 2 2 =4 4 variables: 2 4 =16
8 variables: 2 8 =256 53

W 0
W 1
D 1 = P(W 1 )
D 0 = P(W 0 )
z i = p i (x i1 , ... , x i9 ) , z = p(x) , p=(p 1 , .. p 9 )
Let φ:D 1 →{0,1} , it defines the operator Ψ φ on D o by
Ψ φ (x) = φ(p(x))
The operador Ψ φ is constrained by resolution to D 1
Equivalence classes defined
by
p(x) = p(y)
D 0
54

Properties
Q
Ψ φ
φ
Operators over D 1
Operators over D 0 constrained by resolution on D 1
Operators over D 0
55

D 2
= P(W 2
), D 1
= P(W 1
), D 0
= P(W 0
)
W x ∈D 0
, z ∈D 1
, v ∈D 2
,
0
z i
= p i
(x i,1
, ... , x i,9
) , z = p(x) ,
p=(p 1
, .. p 81
)
v i
= w i
(x i,1
, ... , x i,81
) , v = w(x) ,
p
W 1 w=(w 1
, .. w 9
)
W 2
w The equivalence classes defined by
p may be different by the ones
defined by w.
56

ψ ( x)
=
⎧ ψ0,
N
( x) if
N( x)
> 0
⎪
ψ
,
( x) ( x) = , ( ρ ( x))
>
1 N
if N 0 N
1
0
⎪
⎨ M
⎪ψ m−1,
N
( x) if N( x) = 0,..., N( ρm−2( x)) = 0, N( ρm−
1( x))
> 0
⎪
⎩⎪
ψ
m,
N
( x) if N( x) = 0,..., N( ρm−
1( x)) = 0, N( ρm( x))
> 0
57

Multiresolution Noise
image noise image + noise
61

3x3 window
Pyramid
Restauration
Persisting noise
62

Example
Hybrid Multiresolution Filter: Experiment I
0.14
0.12
W 11
W 10
W 9
W 8
0.1
W 7
W 6
W 5
W 4
0.08
W 3
W 2
W 1
W 0
MAE
0.06
0.04
0.02
0
Ψ11 Ψ10 Ψ9 Ψ8 Ψ7 Ψ6 Ψ5 Ψ4 Ψ3 Ψ2 Ψ1 Ψ0
Window sequence
Single operator
Pyramid operator
64

Envelope constraint

Independent Constraints
Constraints
Independent Constraint
Restriction of
the operators space
K(ψ opt ) ∈ Q ⊆ P(P(W)) (W))
Let be A,BA
⊆ P(W) with A⊆B:
h ψ (x)=1 ∀ x∈A
∀ψ
& h ψ (x)=0∀x∉B,
∀ψ :K(ψ) ∈Q
P(P(W))
K(ψ opt
)
{1}
B
Q
A
{0,1}
Q
B
P(W)
{0}
P(P(W))
A
66

Independent Constraints
Proposition: : if Q is an independent restriction then exist a par of
operators
(α,β)) such that, for any ψ∈Ψ W
K(ψ) ∈Q ⇔ α≤ ψ ≤β
where K(α) ) = A and K(β) ) = B
• All independent constraint is characterized by two operators α and
β
• The pair (α,β)(
) is called “Envelope”
h α
{1}
h β
{1}
A
{0}
A
{1}
B
{0}
B
{0}
P(W)
P(W)
67

Definition
Desing of two heristic filters α and β such that when we
know that α ≤ ψ opt ≤ β, the restriction is defined by:
Q = {ψ : α ≤ ψ ≤ β}
and any filter ψ can be projected into the restriction by
ψenv = ( ψ ∨ α ) ∧ β
68

Definition
β
Operators over
P(W)
ψ
ψ env
Q
α
Operators over P(W)
constrained by envelope
69

Properties
♦ (ψ opt ∨ α ) ∧ β is optimal in Q.
♦If α ≤ ψ opt ≤ β then Error[ψ env ] ≤ Error[ψ]
♦If α ≤ ψ opt ≤ β is not true, then
Lim N→∞ Error[ψ env,N ] > Lim N→∞ Error[ψ N ]
70

Example
71

Noise Edge Detection
Ground
Image
Noise
Addition
Noisy Image
Restoration
Filtered
Image
Edge
Detection
Edge
Detected
Direct Edge Detection
Edge Detected
72

Restoration
a) Machine design of the restoration
ψ pac
designed by examples
b) Human-machine design of the restoration
ψ con
= (ψ pac ∩β) ∪ α
α = δ B⊕B
ε B⊕B
δ B
ε B and β = ε B⊕B
δ B⊕B
ε B
δ B
α and β are alternating sequential filters with
P[ α(S) ≤ I ≤ β(S) ] ≈ 1
B is the 3x3 square
Machine design of
the restoration
Human-Machine
design of the
restoration
0.28 % 0.13 %
0,3
0,25
0,2
0,15
0,1
0,05
Machine design
of the restoration
Human design of
the restoration
0
Error (%)
73

Noise Edge Detection
Edge
Detection
a) Machine design over noisy images
ζ pac
designed by examples from noisy images
b) Human design after restoration
ζ = I d
- ε B
B is the 3x3 square
c) Machine design after restoration
ζ pac
designed by examples from restored images
Machine
design over
noisy images
Human
design after
restoration
Machine
design after
restoration
0.65 % 0.27 % 0.24 %
0,7
0,6
0,5
0,4
0,3
0,2
0,1
0
Error (%)
Machine design over
noisy images
Human design after
restoration
Machine design after
restoration
74

Noise Edge Detection
Machine design over
noisy images
Error = 0.65%
Human design after
restoration
Error = 0.27%
Machine design after
restoration
Error = 0.24%
75

Envelope multi-resolution
constraint

Definition
♦ W 1 ⊂ W 0 , ρ:D 0 → D 1 is a resolution mapping
♦ α, β: D 1 → {0,1} with α ≤ β
♦ ψ : D 0 → {0.1}
ψ ρ
( x)
=
⎧ 1
⎪
⎨ 0
⎪
⎩ψ
( x)
ifα(
ρ(
x))
if β ( ρ(
x))
= 1
= 0
otherwhise
♦ ψ ρ = (ψ ∧ β’) ∨ α’ , α’(x) = α(r(x)) and β’(x) = β(r(x))
77

Definition
Ψ φ
β’
φ
β
φ env
ψ ρ
Q
α
D 0
α’
D 1
78

Teorema
β’
β
φ
ψ
φ env
ψ ρ
α
Q
D
α’
1
D 0
79

Piramidal Design:
Let ψ i,env,N = ( ψ i,N ∧ β’) ∨ α’, be the projection of the resolution
constrained filter inside the envelope (α’, β’)
ψ
env−mres
( x)
=
⎧ ψ0,
N
( x) if
N( x)
> 0
⎪
ψ
env N
if N = N ρ >
1, ,
( x) ( x) 0, (
1( x))
0
⎪
⎨ M
⎪ψ m−1, env,
N
( x) if N( x) = 0,..., N( ρm−2( x)) = 0, N( ρm−
1( x))
> 0
⎪
⎩⎪
ψ
m, env,
N
( x) if N( x) = 0,..., N( ρm−
1( x)) = 0, N( ρm( x))
> 0
80

Properties:
♦ ψ env-mres is a consistent estimator of ψ opt
♦ If the envelope is well defined on D 1 , then the ρ-envelope of a
resolution constrained filter is advantageous
81

B
E
α = ε B (γ E φ E γ E )
β = δ B (φ E γ E φ E )
82

Gray-scale operator design:
aperture

Spatial Translation Invariance
85

Gray-scale Translation Invariance
86

Locally defined in W
Ψ
( f )( x)
= Ψ( f / W )(
x)
x
87

Locally defined in W and K
( u / K )( z)
= ∧{
∨{
−k,
u(
z)
− y},
k}
y
88

Aperture Operator
W K = { − 2,
−1,0,1,2
}
30
25
2
1
-2 1 2 2 2
-2 1 2 2 2
20
β ψ
0
-2 1 2 2 2
15
-1
-2
-2 1 1 1 1
-2 -2 -2 -2 -2
-2 -1 0 1 2
10
5
0
input
output
89

2
-2 1 2 2 2
Aperture
β ψ
1
0
-2 1 2 2 2
-2 1 2 2 2
Operator
-1
-2 1 1 1 1
-2
-2 -2 -2 -2 -2
-2 -1 0 1 2
ψ
u(o)
βψ
14
12 13 14 15 16
14
10 11 12 13 14
14
2 2 2 2 2
13
12 13 14 15 15
13
10 11 12 13 14
13
2 2 2 2 1
12
11
12 13 14 14 12
12 13 13 11 12
=
12
11
10 11 12 13 14
10 11 12 13 14
+
12
11
2 2 2 1 -2
2 2 1 -2 -2
10
12 12 10 11 12
10
10 11 12 13 14
10
2 1 -2 -2 -2
10 11 12 13 14
10 11 12 13 14
10 11 12 13 14
90

Let a, b ∈ Fun[W,L
W,L], a ≤ b iff a(x) ≤ b(x),
x ∈ W
|W| = 2
a
b
Interval [a,b]] = {u{
∈ Fun[W,L
W,L]:
a ≤ u ≤ b}
91

Sup-generating operator:
λ
( u) = 1 ⇔ u ∈[
a,
]
a, b
b
[a,b]
0 0 0 0 0
0 0 1 1 1
0 0 1 1 1
0 0 1 1 1
0 0 0 0 0
λ a,b
92

Kernel of ψ at y: K(ψ)(y) = {u ∈ Fun[W,L]: y ≤ ψ(u)}
2
1
0
-1
-2
0 1 2 2 2
0 1 2 2 2
-1 1 2 2 2
-1 1 1 1 1
-2 -1 -1 -1 -1
-2 -1 0 1 2
K(ψ)(-2)
K(ψ)(-1) K(ψ)(0) K(ψ)(1)
K(ψ)(2)
93

Basis of ψ at y: B(ψ) is the set of maximal intervals contained in K(ψ)
K(ψ)(-2)
K(ψ)(-1) K(ψ)(0) K(ψ)(1)
K(ψ)(2)
B(ψ)(-2)
B(ψ)(-1) B(ψ)(0) B(ψ)(1)
B(ψ)(2)
94

Sup-representation
( u) = U{ y ∈ M : { ( ) :[ , ] ( )( )}
1}
,
u a b ∈ B y =
ψ U λ a
ψ
b
K(ψ)(-2)
K(ψ)(-1) K(ψ)(0) K(ψ)(1)
K(ψ)(2)
ψ(-1,-1) = 1
95

Observed
Ideal
These are part of the observed and ideal images (512x512)
96

MAE x Number of Examples
97

Debluring - Aperture x Optimal
linear
Aperture 17p x 5 x 5
Optimal linear 7x7
98

Resolution Enhancement
99

Resolution Enhancement
(0,0) (0,1)
Ψ 2
Ψ 0
Ψ 1
Ψ 3
Ψ 3
100

Resolution Enhancement - Results
Original
Aperture: 3x3x21x51
Linear
Bilinear
101

Resolution Enhancement - Results
Zoom
Original
Aperture: 3x3x21x51
Linear
Bilinear
102

Gray-scale operator design:
stack filters

A stack filter is a gray-scale operator characterized
by a positive (i.e., increasing) Boolean function
ψ ( f ) = max{ t ∈ K : ψ ( T [ f ]) = 1}
t
where
T t
[ f ] = { x ∈W
: f ( x)
≥ t}
104

Impulse noise removal (1)
training images
105

Impulse noise removal (2)
test image iteration 1
106

Impulse noise removal (3)
test image iteration 5
107

Robustness (1)
test image iteration 1
108

Robustness (2)
test image iteration 5
109

Conclusion
Design of Morphological Operators: a
discrete nature problem
Fundamentals: Algebra, Statistics,
Combinatory
Real problems solution
Design techniques adequate to introduce
prior knowledge
Identification of Lattice Dynamical Systems
110