Template tracking methods
Template tracking methods Template tracking methods
Technischen Universität München Winter Semester 2009/2010 TRACKING and DETECTION in COMPUTER VISION Template tracking methods Slobodan Ilić
- Page 2 and 3: Template based-tracking • Energy-
- Page 4 and 5: Template warping Template Tracking
- Page 6 and 7: Template warping Given the template
- Page 8 and 9: Template warping Given the template
- Page 10 and 11: The tracking task The goal of templ
- Page 12 and 13: The tracking task The goal of templ
- Page 14 and 15: Assumptions • No errors in the te
- Page 16 and 17: Assumptions • No errors in the te
- Page 18 and 19: The derivation of Lucas-Kanade Line
- Page 20 and 21: The derivation of Lucas-Kanade Line
- Page 22 and 23: Typical warps • Translation, like
- Page 24 and 25: The LK alg. summary Template Tracki
- Page 26 and 27: • • • • Compositional Image
- Page 28 and 29: • • • Derivation After applyi
- Page 30 and 31: Compositional vs. Additive Differen
- Page 32 and 33: Derivation • After first order Ta
- Page 34 and 35: ESM (Efficient Second-order Method)
- Page 36 and 37: ESM (Efficient Second-order Method)
- Page 38 and 39: ESM (Efficient Second-order Method)
- Page 40 and 41: ESM (Efficient Second-order Method)
- Page 42 and 43: ESM (Efficient Second-order Method)
- Page 44 and 45: ESM (Efficient Second-order Method)
- Page 46 and 47: ESM Convergence Gauss-Newton ESM Te
- Page 48 and 49: • • • • Hyperplane approxim
- Page 50 and 51: Advantages Can "jump" over local m
Technischen Universität München<br />
Winter Semester 2009/2010<br />
TRACKING and DETECTION in <br />
COMPUTER VISION<br />
<strong>Template</strong> <strong>tracking</strong> <strong>methods</strong><br />
Slobodan Ilić
<strong>Template</strong> based-<strong>tracking</strong><br />
• Energy-based <strong>methods</strong><br />
• The Lucas-Kanade(LK) algorithm<br />
• Compositional Algorithm<br />
• Inverse Compositional Algorithm (IC)<br />
• Efficient Second Order Method (ESM)<br />
• Learning <strong>methods</strong><br />
• Learning linear predictors(Hyperplane template<br />
matching of Jurie-Dhome)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
2
template image<br />
Motivation<br />
Follow a template image in an<br />
image sequence by estimating<br />
the warp.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
3
<strong>Template</strong> warping<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
4
<strong>Template</strong> warping<br />
Given the template which we want to track:<br />
<strong>Template</strong> image<br />
T (x)<br />
T<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
4
<strong>Template</strong> warping<br />
Given the template which we want to track:<br />
• Take all pixels form the template and<br />
<strong>Template</strong> image<br />
T (x)<br />
x<br />
T<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
4
<strong>Template</strong> warping<br />
Given the template which we want to track:<br />
• Take all pixels form the template and<br />
• Warp them using the function parameterized in<br />
terms of parameters p<br />
to the input image<br />
<strong>Template</strong> image<br />
T (x)<br />
x<br />
T<br />
warp<br />
W (x; p)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
4<br />
W (x; p)<br />
Input image
<strong>Template</strong> warping<br />
Given the template which we want to track:<br />
• Take all pixels form the template and<br />
• Warp them using the function parameterized in<br />
terms of parameters p<br />
to the input image<br />
• Assign the pixel value of the input image at the warped<br />
location to the template image<br />
<strong>Template</strong> image<br />
T (x)<br />
x<br />
T<br />
warp<br />
W (x; p)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
4<br />
W (x; p)<br />
Input image<br />
I(W (x; p))
The <strong>tracking</strong> task<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
5
The <strong>tracking</strong> task<br />
The goal of template-based <strong>tracking</strong> is to find the<br />
parameters p such that:<br />
I(W(x; p)) = T (x)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
5
The <strong>tracking</strong> task<br />
The goal of template-based <strong>tracking</strong> is to find the<br />
parameters p such that:<br />
I(W(x; p)) = T (x)<br />
Using a prediction ˆp as an approximation of ¯p we can reformulate<br />
the goal:<br />
• Find the parameter increments ∆p such that:<br />
¯p ← ˆp + ∆p<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
5
The <strong>tracking</strong> task<br />
The goal of template-based <strong>tracking</strong> is to find the<br />
parameters p such that:<br />
• by minimizing the norm of:<br />
I(W(x; p)) = T (x)<br />
Using a prediction ˆp as an approximation of ¯p we can reformulate<br />
the goal:<br />
• Find the parameter increments ∆p such that:<br />
¯p ← ˆp + ∆p<br />
This is a non-linear minimization problem.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
5<br />
(1)
Assumptions<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
6
Assumptions<br />
• No errors in the template image<br />
boundaries:<br />
only the appearance of the object to be<br />
tracked appears in the template image.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
6
Assumptions<br />
• No errors in the template image<br />
boundaries:<br />
only the appearance of the object to be<br />
tracked appears in the template image.<br />
• No occlusion: the entire template is<br />
visible in the input image.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
6
Assumptions<br />
• No errors in the template image<br />
boundaries:<br />
only the appearance of the object to be<br />
tracked appears in the template image.<br />
• No occlusion: the entire template is<br />
visible in the input image.<br />
• Brightness constancy assumption: the<br />
intensity of the object appearance is<br />
always the same.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
6
The Lucas-Kanade<br />
Uses the Gauss-Newton method for<br />
minimization:<br />
• Applies a first-order approximation<br />
• Minimizes iteratively<br />
The warp has to be differentiable.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
7
The derivation<br />
of Lucas-Kanade<br />
Linearize the error function of Eq. (1) on slide 5 by using a<br />
first-order Taylor series approximation of<br />
warped source image :<br />
And minimize the following function:<br />
=<br />
(2)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
8
The derivation<br />
of Lucas-Kanade<br />
Linearize the error function of Eq. (1) on slide 5 by using a<br />
first-order Taylor series approximation of<br />
warped source image :<br />
And minimize the following function:<br />
Image gradient evaluated at W(x; p)<br />
=<br />
(2)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
8
The derivation<br />
of Lucas-Kanade<br />
Linearize the error function of Eq. (1) on slide 5 by using a<br />
first-order Taylor series approximation of<br />
warped source image :<br />
And minimize the following function:<br />
=<br />
(2)<br />
Image gradient evaluated at W(x; p) Jacobian of the warp<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
8
Jacobian of the warp<br />
If warp is a function:<br />
then its Jacobian is:<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
9
Typical warps<br />
• Translation, like in optical-flow:<br />
• Affine, like in template <strong>tracking</strong>:<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
10
LK derivation<br />
Minimizing the Eq. 2 from slide 8 is a least-square problem<br />
and has a closed form solution. The partial derivatives of<br />
Eq.2 in respect with the parameter update is:<br />
setting equation to zero gives closed-form solution:<br />
where H is nxn (Gauss-Newton) approximation of the<br />
Hessian matrix:<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
11<br />
∆p
The LK alg. summary<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
12<br />
Iterate<br />
until ∆p < ɛ
Alternative approaches<br />
Lucas-Kanade approximately minimizes:<br />
with respect to ∆pand<br />
updates the parameters is step 9<br />
like ¯p ← ˆp + ∆p<br />
BUT, these are not the only ways to minimize this function,<br />
thus 3 other approaches are introduced:<br />
• Compositional Image Alignment<br />
• Inverse Compositional Image Alignment<br />
• Inverse Additive Image Alignment<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
13
•<br />
•<br />
•<br />
•<br />
Compositional Image<br />
Alignment<br />
The warp is composed of two warps:<br />
which can be written as:<br />
and the solution is sought for the warp<br />
instead of additive update of the parameters<br />
Thus the problem to minimize becomes:<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
14<br />
∆p<br />
(3)
Compositional Affine Warp<br />
• Affine warp:<br />
• Compositional affine wrap:<br />
W(W(x; ∆p); p)) =<br />
<br />
1+p1 p3 p5<br />
p2 1 + p4 p6<br />
⎛<br />
⎝ 1+∆p1<br />
⎞ ⎛<br />
∆p3 ∆p5<br />
∆p2 1 + ∆p4 ∆p6⎠<br />
⎝<br />
0 0 1<br />
x<br />
⎞<br />
y⎠<br />
1<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
15
•<br />
•<br />
•<br />
Derivation<br />
After applying first order Taylor approximation to<br />
I(W(x; ∆p); p))<br />
the Eq. 3 from slide 14 becomes:<br />
where I(W)(x) denotes the warped image I(W(x; p))<br />
and in order to proceed we assume that the warp<br />
W(x; 0) =x<br />
simplifies to:<br />
is identity warp, thus the above eq.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
16
•<br />
•<br />
Compositional vs.<br />
Additive Approach<br />
Forward additive approach:<br />
Compute image gradient and warp it with W(x; p)<br />
Compositional approach:<br />
Compute image gradient of<br />
the warped image I(W) Jacobian of the warp is<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
17<br />
Jacobian of the warp<br />
is evaluated at (x; p)<br />
evaluated at (x; 0)<br />
and is constant
Compositional vs. Additive<br />
Differences<br />
• The gradient of I(W(x; p))<br />
should be used in<br />
step (3) of the algorithm from slide 12.<br />
• The Jacobian of the warp can be pre-computed<br />
because it is evaluated at (x;0) rather then being<br />
recomputed at each iteration in step (4).<br />
• The warp is updated<br />
in step 9 and not warp parameters.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
18
Inverse Compositional (IC)<br />
T(x)<br />
W(x; ∆p)<br />
<strong>Template</strong> image<br />
Approach<br />
W(x; p) ← W(x; p) ◦ W(x; ∆p) −1<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
19<br />
(4)<br />
W(x; p)<br />
Source image
Derivation<br />
• After first order Taylor expansion Eq. 4 becomes:<br />
• and assuming that W(x; 0) =x<br />
is the identity warp the<br />
solution is:<br />
• where the Hessian of image is replaced by the one of the<br />
template:<br />
• since the there is nothing in H which depends on p it is<br />
constant and can be pre-computed<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
20
IC Algorithm<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
21
ESM (Efficient Second-order<br />
Method)<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]<br />
Like Gauss-Newton but replace J p=0 by ½(J p=0 + J p=dp).<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM (Efficient Second-order<br />
Method)<br />
(1) I = T + J p=0dp + dp T H p=0dp [second-order Taylor expansion]<br />
(2) J p=dp = J p=0 + 2dp T H p=0 [derivation of (1) wrt p]<br />
(3) dp T H p=0 = ½(J p=dp - J p=0) [from Equation (2)]<br />
(4) I = T + J p=0 + ½(J p=dp - J p=0)dp [by injecting (3) in (1)]<br />
(5) dp = [½(J p=0 + J p=dp)] + (I - T) [from Equation (4)]<br />
Like Gauss-Newton but replace J p=0 by ½(J p=0 + J p=dp).<br />
Need to compute J p=dp at each iteration, and a pseudo-inverse at each<br />
iteration, but need much less iterations.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
22
ESM Convergence<br />
Gauss-Newton ESM<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
23
Learning a Linear Predictor<br />
• From the IC approach we had:<br />
• Where we can denote Jɛ = ∇T to be<br />
Jacobian of the error function we minimize<br />
∂W<br />
∂p<br />
• Thus we can rewrite it:<br />
• or<br />
[Jurie&Dhome PAMI02]<br />
∆p =(J T ɛ (0)Jɛ(0)) −1 J T ɛ (0)ɛ<br />
∆p = Aɛ<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
24
•<br />
•<br />
•<br />
•<br />
Hyperplane<br />
approximation<br />
If for each pixel we write an error function:<br />
yi(∆p) =T(W(xi; ∆p)) − I(W(xi; p)<br />
and the total error function is<br />
with the solution being:<br />
we can rewrite it like:<br />
∆p = Aɛ<br />
we see that a11, . . . , aij . . . anN are coefficients of n hyperplanes<br />
that can be estimated using linear least-squares<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
25<br />
E = <br />
i<br />
yi(∆p) 2<br />
∆p1<br />
∆pj<br />
∆pN
Training<br />
• Can we learn A and make computation even<br />
faster?<br />
A is computed offline by regression:<br />
{( , ) , ( , ) , ... ,( , )}<br />
∆p0<br />
∆p1<br />
∆pNp<br />
y1(0) y2(0) image differences<br />
N<br />
by minimizing: (∆pk − Ayk(0)) 2<br />
By writing<br />
k=1<br />
and assuming N > n<br />
=⇒<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
26<br />
yNp(0)<br />
H =(yi(0), . . . , yN(0)) Y =(∆p1, . . . , ∆pN)<br />
Y = AhH Ah = YH T (HH T ) −1
Advantages<br />
Can "jump" over local minimums.<br />
Handle faster motion.<br />
O<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
27<br />
p
Active Shape Models<br />
For deformable objects:<br />
(u i, v i)<br />
€<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
28<br />
p =<br />
⎡ x⎤<br />
⎢ ⎥<br />
⎢<br />
tu⎥ ⎢ t ⎥ v<br />
⎢ ⎥<br />
⎢<br />
θ<br />
⎥<br />
⎣ ⎢ s⎦<br />
⎥<br />
with x =<br />
⎡<br />
⎢<br />
⎢<br />
⎢<br />
⎢<br />
⎣<br />
<br />
u i<br />
v i<br />
<br />
⎤<br />
⎥<br />
⎥<br />
⎥<br />
⎥<br />
⎦<br />
(t u, t v): 2D translation;<br />
ϑ: 2D rotation;<br />
s: scale.
Dimensionality Reduction<br />
Training set:<br />
{ x } i<br />
Covariance matrix of the training data: Cov = ( x i − x ) ( x i − x ) T<br />
⎛ λ1 ⎜<br />
λ2 SVD of the covariance matrix: Cov = U⎜<br />
⎜<br />
⎜<br />
⎝<br />
Keep only the first columns of U : U = ( Rx ).<br />
x = x + R xc<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
29<br />
∑<br />
i<br />
<br />
⎞<br />
⎟<br />
⎟<br />
⎟<br />
⎟<br />
λn⎠ U T<br />
U 2<br />
c<br />
U 1
First Modes<br />
€<br />
Can constrain with c i < 3 λ i<br />
x = x + R x c<br />
Can be considered<br />
as a linear warp:<br />
W(x; c) =¯x + Rxc<br />
c are parameter of<br />
the warp we are<br />
searching<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
30
Fit Function<br />
min<br />
c,t u ,t v ,θ ,s<br />
∑<br />
i<br />
dist( Tr ( x ),y )<br />
tu ,tv ,θ ,s i i<br />
Optimum can be found with linear algebra operations.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
31
Convergence<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
32
Active Appearance<br />
€<br />
g<br />
Models<br />
g = g + R gc<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
33
min<br />
c,t u ,tv ,θ ,s Tr ( tu ,tv ,θ ,s g( c)<br />
) − I 2<br />
Fit Function<br />
= min<br />
c,t u ,t v ,θ ,s<br />
( )<br />
[ Tr ( tu ,tv ,θ ,s g( c)<br />
) ] − Ii i<br />
Non-linear least-squares.<br />
Optimization can be performed with the Gauss-Newton algorithm.<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
34<br />
∑<br />
i<br />
2
Convergence<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
35
Example<br />
http://www.ri.cmu.edu/research_project_detail.html?project_id=448&menu_id=261<br />
<strong>Template</strong> Tracking Methods Ilic Slobodan<br />
36