Template tracking methods

Technischen
Universität
München
Winter
Semester
2009/2010
TRACKING
and
DETECTION
in
COMPUTER
VISION
Template
tracking
methods
Slobodan
Ilić

Template based-tracking
• Energy-based methods
• The Lucas-Kanade(LK) algorithm
• Compositional Algorithm
• Inverse Compositional Algorithm (IC)
• Efficient Second Order Method (ESM)
• Learning methods
• Learning linear predictors(Hyperplane template
matching of Jurie-Dhome)
Template Tracking Methods Ilic Slobodan
2

template image
Motivation
Follow a template image in an
image sequence by estimating
the warp.
Template Tracking Methods Ilic Slobodan
3

Template warping
Template Tracking Methods Ilic Slobodan
4

Template warping
Given the template which we want to track:
Template image
T (x)
T
Template Tracking Methods Ilic Slobodan
4

Template warping
Given the template which we want to track:
• Take all pixels form the template and
Template image
T (x)
x
T
Template Tracking Methods Ilic Slobodan
4

Template warping
Given the template which we want to track:
• Take all pixels form the template and
• Warp them using the function parameterized in
terms of parameters p
to the input image
Template image
T (x)
x
T
warp
W (x; p)
Template Tracking Methods Ilic Slobodan
4
W (x; p)
Input image

Template warping
Given the template which we want to track:
• Take all pixels form the template and
• Warp them using the function parameterized in
terms of parameters p
to the input image
• Assign the pixel value of the input image at the warped
location to the template image
Template image
T (x)
x
T
warp
W (x; p)
Template Tracking Methods Ilic Slobodan
4
W (x; p)
Input image
I(W (x; p))

The tracking task
Template Tracking Methods Ilic Slobodan
5

The tracking task
The goal of template-based tracking is to find the
parameters p such that:
I(W(x; p)) = T (x)
Template Tracking Methods Ilic Slobodan
5

The tracking task
The goal of template-based tracking is to find the
parameters p such that:
I(W(x; p)) = T (x)
Using a prediction ˆp as an approximation of ¯p we can reformulate
the goal:
• Find the parameter increments ∆p such that:
¯p ← ˆp + ∆p
Template Tracking Methods Ilic Slobodan
5

The tracking task
The goal of template-based tracking is to find the
parameters p such that:
• by minimizing the norm of:
I(W(x; p)) = T (x)
Using a prediction ˆp as an approximation of ¯p we can reformulate
the goal:
• Find the parameter increments ∆p such that:
¯p ← ˆp + ∆p
This is a non-linear minimization problem.
Template Tracking Methods Ilic Slobodan
5
(1)

Assumptions
Template Tracking Methods Ilic Slobodan
6

Assumptions
• No errors in the template image
boundaries:
only the appearance of the object to be
tracked appears in the template image.
Template Tracking Methods Ilic Slobodan
6

Assumptions
• No errors in the template image
boundaries:
only the appearance of the object to be
tracked appears in the template image.
• No occlusion: the entire template is
visible in the input image.
Template Tracking Methods Ilic Slobodan
6

Assumptions
• No errors in the template image
boundaries:
only the appearance of the object to be
tracked appears in the template image.
• No occlusion: the entire template is
visible in the input image.
• Brightness constancy assumption: the
intensity of the object appearance is
always the same.
Template Tracking Methods Ilic Slobodan
6

The Lucas-Kanade
Uses the Gauss-Newton method for
minimization:
• Applies a first-order approximation
• Minimizes iteratively
The warp has to be differentiable.
Template Tracking Methods Ilic Slobodan
7

The derivation
of Lucas-Kanade
Linearize the error function of Eq. (1) on slide 5 by using a
first-order Taylor series approximation of
warped source image :
And minimize the following function:
=
(2)
Template Tracking Methods Ilic Slobodan
8

The derivation
of Lucas-Kanade
Linearize the error function of Eq. (1) on slide 5 by using a
first-order Taylor series approximation of
warped source image :
And minimize the following function:
Image gradient evaluated at W(x; p)
=
(2)
Template Tracking Methods Ilic Slobodan
8

The derivation
of Lucas-Kanade
Linearize the error function of Eq. (1) on slide 5 by using a
first-order Taylor series approximation of
warped source image :
And minimize the following function:
=
(2)
Image gradient evaluated at W(x; p) Jacobian of the warp
Template Tracking Methods Ilic Slobodan
8

Jacobian of the warp
If warp is a function:
then its Jacobian is:
Template Tracking Methods Ilic Slobodan
9

Typical warps
• Translation, like in optical-flow:
• Affine, like in template tracking:
Template Tracking Methods Ilic Slobodan
10

LK derivation
Minimizing the Eq. 2 from slide 8 is a least-square problem
and has a closed form solution. The partial derivatives of
Eq.2 in respect with the parameter update is:
setting equation to zero gives closed-form solution:
where H is nxn (Gauss-Newton) approximation of the
Hessian matrix:
Template Tracking Methods Ilic Slobodan
11
∆p

The LK alg. summary
Template Tracking Methods Ilic Slobodan
12
Iterate
until ∆p < ɛ

Alternative approaches
Lucas-Kanade approximately minimizes:
with respect to ∆pand
updates the parameters is step 9
like ¯p ← ˆp + ∆p
BUT, these are not the only ways to minimize this function,
thus 3 other approaches are introduced:
• Compositional Image Alignment
• Inverse Compositional Image Alignment
• Inverse Additive Image Alignment
Template Tracking Methods Ilic Slobodan
13

•
•
•
•
Compositional Image
Alignment
The warp is composed of two warps:
which can be written as:
and the solution is sought for the warp
instead of additive update of the parameters
Thus the problem to minimize becomes:
Template Tracking Methods Ilic Slobodan
14
∆p
(3)

Compositional Affine Warp
• Affine warp:
• Compositional affine wrap:
W(W(x; ∆p); p)) =
1+p1 p3 p5
p2 1 + p4 p6
⎛
⎝ 1+∆p1
⎞ ⎛
∆p3 ∆p5
∆p2 1 + ∆p4 ∆p6⎠
⎝
0 0 1
x
⎞
y⎠
1
Template Tracking Methods Ilic Slobodan
15

•
•
•
Derivation
After applying first order Taylor approximation to
I(W(x; ∆p); p))
the Eq. 3 from slide 14 becomes:
where I(W)(x) denotes the warped image I(W(x; p))
and in order to proceed we assume that the warp
W(x; 0) =x
simplifies to:
is identity warp, thus the above eq.
Template Tracking Methods Ilic Slobodan
16

•
•
Compositional vs.
Additive Approach
Forward additive approach:
Compute image gradient and warp it with W(x; p)
Compositional approach:
Compute image gradient of
the warped image I(W) Jacobian of the warp is
Template Tracking Methods Ilic Slobodan
17
Jacobian of the warp
is evaluated at (x; p)
evaluated at (x; 0)
and is constant

Compositional vs. Additive
Differences
• The gradient of I(W(x; p))
should be used in
step (3) of the algorithm from slide 12.
• The Jacobian of the warp can be pre-computed
because it is evaluated at (x;0) rather then being
recomputed at each iteration in step (4).
• The warp is updated
in step 9 and not warp parameters.
Template Tracking Methods Ilic Slobodan
18

Inverse Compositional (IC)
T(x)
W(x; ∆p)
Template image
Approach
W(x; p) ← W(x; p) ◦ W(x; ∆p) −1
Template Tracking Methods Ilic Slobodan
19
(4)
W(x; p)
Source image

Derivation
• After first order Taylor expansion Eq. 4 becomes:
• and assuming that W(x; 0) =x
is the identity warp the
solution is:
• where the Hessian of image is replaced by the one of the
template:
• since the there is nothing in H which depends on p it is
constant and can be pre-computed
Template Tracking Methods Ilic Slobodan
20

IC Algorithm
Template Tracking Methods Ilic Slobodan
21

ESM (Efficient Second-order
Method)
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]
Like Gauss-Newton but replace J p=0 by ½(J p=0 + J p=dp).
Template Tracking Methods Ilic Slobodan
22

ESM (Efficient Second-order
Method)
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]
Like Gauss-Newton but replace J p=0 by ½(J p=0 + J p=dp).
Need to compute J p=dp at each iteration, and a pseudo-inverse at each
iteration, but need much less iterations.
Template Tracking Methods Ilic Slobodan
22

ESM Convergence
Gauss-Newton ESM
Template Tracking Methods Ilic Slobodan
23

Learning a Linear Predictor
• From the IC approach we had:
• Where we can denote Jɛ = ∇T to be
Jacobian of the error function we minimize
∂W
∂p
• Thus we can rewrite it:
• or
[Jurie&Dhome PAMI02]
∆p =(J T ɛ (0)Jɛ(0)) −1 J T ɛ (0)ɛ
∆p = Aɛ
Template Tracking Methods Ilic Slobodan
24

•
•
•
•
Hyperplane
approximation
If for each pixel we write an error function:
yi(∆p) =T(W(xi; ∆p)) − I(W(xi; p)
and the total error function is
with the solution being:
we can rewrite it like:
∆p = Aɛ
we see that a11, . . . , aij . . . anN are coefficients of n hyperplanes
that can be estimated using linear least-squares
Template Tracking Methods Ilic Slobodan
25
E =
i
yi(∆p) 2
∆p1
∆pj
∆pN

Training
• Can we learn A and make computation even
faster?
A is computed offline by regression:
{( , ) , ( , ) , ... ,( , )}
∆p0
∆p1
∆pNp
y1(0) y2(0) image differences
N
by minimizing: (∆pk − Ayk(0)) 2
By writing
k=1
and assuming N > n
=⇒
Template Tracking Methods Ilic Slobodan
26
yNp(0)
H =(yi(0), . . . , yN(0)) Y =(∆p1, . . . , ∆pN)
Y = AhH Ah = YH T (HH T ) −1

Advantages
Can "jump" over local minimums.
Handle faster motion.
O
Template Tracking Methods Ilic Slobodan
27
p

Active Shape Models
For deformable objects:
(u i, v i)
€
Template Tracking Methods Ilic Slobodan
28
p =
⎡ x⎤
⎢ ⎥
⎢
tu⎥ ⎢ t ⎥ v
⎢ ⎥
⎢
θ
⎥
⎣ ⎢ s⎦
⎥
with x =
⎡
⎢
⎢
⎢
⎢
⎣
u i
v i
⎤
⎥
⎥
⎥
⎥
⎦
(t u, t v): 2D translation;
ϑ: 2D rotation;
s: scale.

Dimensionality Reduction
Training set:
{ x } i
Covariance matrix of the training data: Cov = ( x i − x ) ( x i − x ) T
⎛ λ1 ⎜
λ2 SVD of the covariance matrix: Cov = U⎜
⎜
⎜
⎝
Keep only the first columns of U : U = ( Rx ).
x = x + R xc
Template Tracking Methods Ilic Slobodan
29
∑
i
⎞
⎟
⎟
⎟
⎟
λn⎠ U T
U 2
c
U 1

First Modes
€
Can constrain with c i < 3 λ i
x = x + R x c
Can be considered
as a linear warp:
W(x; c) =¯x + Rxc
c are parameter of
the warp we are
searching
Template Tracking Methods Ilic Slobodan
30

Fit Function
min
c,t u ,t v ,θ ,s
∑
i
dist( Tr ( x ),y )
tu ,tv ,θ ,s i i
Optimum can be found with linear algebra operations.
Template Tracking Methods Ilic Slobodan
31

Convergence
Template Tracking Methods Ilic Slobodan
32

Active Appearance
€
g
Models
g = g + R gc
Template Tracking Methods Ilic Slobodan
33

min
c,t u ,tv ,θ ,s Tr ( tu ,tv ,θ ,s g( c)
) − I 2
Fit Function
= min
c,t u ,t v ,θ ,s
( )
[ Tr ( tu ,tv ,θ ,s g( c)
) ] − Ii i
Non-linear least-squares.
Optimization can be performed with the Gauss-Newton algorithm.
Template Tracking Methods Ilic Slobodan
34
∑
i
2

Convergence
Template Tracking Methods Ilic Slobodan
35

Example
http://www.ri.cmu.edu/research_project_detail.html?project_id=448&menu_id=261
Template Tracking Methods Ilic Slobodan
36