Separable Nonlinear Least Squares Problems in Image Processing
Separable Nonlinear Least Squares Problems in Image Processing
Separable Nonlinear Least Squares Problems in Image Processing
You also want an ePaper? Increase the reach of your titles
YUMPU automatically turns print PDFs into web optimized ePapers that Google loves.
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
<strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong><br />
<strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g<br />
Collaborators:<br />
Julianne Chung and James Nagy<br />
Emory University<br />
Atlanta, GA, USA<br />
Eldad Haber (Emory)<br />
Per Christian Hansen (Tech. Univ. of Denmark)<br />
Dianne O’Leary (University of Maryland)<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Inverse <strong>Problems</strong> <strong>in</strong> Imag<strong>in</strong>g<br />
Imag<strong>in</strong>g problems are often modeled as:<br />
b = Ax + e<br />
where<br />
A - large, ill-conditioned matrix<br />
b - known, measured (image) data<br />
e - noise, statistical properties may be known<br />
Goal: Compute approximation of image x<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Inverse <strong>Problems</strong> <strong>in</strong> Imag<strong>in</strong>g<br />
A more realistic image formation model is:<br />
where<br />
b = A(y) x + e<br />
A(y) - large, ill-conditioned matrix<br />
b - known, measured (image) data<br />
e - noise, statistical properties may be known<br />
y - parameters def<strong>in</strong><strong>in</strong>g A, usually approximated<br />
Goal: Compute approximation of image x<br />
and improve estimate of parameters y<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
b = A(y) x + e = observed image<br />
where y describes blurr<strong>in</strong>g function<br />
Given: b and an estimate of y<br />
Standard <strong>Image</strong> Deblurr<strong>in</strong>g:<br />
Compute approximation of x<br />
Better approach:<br />
Jo<strong>in</strong>tly improve estimate of y<br />
and compute approximation of x.<br />
Observed <strong>Image</strong><br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
b = A(y) x + e = observed image<br />
where y describes blurr<strong>in</strong>g function<br />
Given: b and an estimate of y<br />
Standard <strong>Image</strong> Deblurr<strong>in</strong>g:<br />
Compute approximation of x<br />
Better approach:<br />
Jo<strong>in</strong>tly improve estimate of y<br />
and compute approximation of x.<br />
Observed <strong>Image</strong><br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
b = A(y) x + e = observed image<br />
where y describes blurr<strong>in</strong>g function<br />
Given: b and an estimate of y<br />
Standard <strong>Image</strong> Deblurr<strong>in</strong>g:<br />
Compute approximation of x<br />
Better approach:<br />
Jo<strong>in</strong>tly improve estimate of y<br />
and compute approximation of x.<br />
Reconstruction us<strong>in</strong>g <strong>in</strong>itial PSF<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
b = A(y) x + e = observed image<br />
where y describes blurr<strong>in</strong>g function<br />
Given: b and an estimate of y<br />
Standard <strong>Image</strong> Deblurr<strong>in</strong>g:<br />
Compute approximation of x<br />
Better approach:<br />
Jo<strong>in</strong>tly improve estimate of y<br />
and compute approximation of x.<br />
Reconstruction after 8 GN iterations<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
1−th low resolution image<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
8−th low resolution image<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
15−th low resolution image<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
22−th low resolution image<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
29−th low resolution image<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
⎡ ⎤ ⎡ ⎤ ⎡ ⎤<br />
b 1 A(y 1 ) e 1<br />
⎢ ⎥ ⎢ ⎥ ⎢ ⎥<br />
⎣ . ⎦ = ⎣ . ⎦ x+ ⎣ . ⎦<br />
b m A(y m ) e m<br />
} {{ } } {{ } } {{ }<br />
29−th low resolution image<br />
b = A(y) x + e<br />
y = registration, blurr<strong>in</strong>g, etc.,<br />
parameters<br />
Goal: Improve parameters y and<br />
compute x<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Application: <strong>Image</strong> Data Fusion<br />
b j = A(y j ) x + e j<br />
(collected low resolution images)<br />
⎡ ⎤ ⎡ ⎤ ⎡ ⎤<br />
b 1 A(y 1 ) e 1<br />
⎢ ⎥ ⎢ ⎥ ⎢ ⎥<br />
⎣ . ⎦ = ⎣ . ⎦ x+ ⎣ . ⎦<br />
b m A(y m ) e m<br />
} {{ } } {{ } } {{ }<br />
Reconstructed high resolution image<br />
b = A(y) x + e<br />
y = registration, blurr<strong>in</strong>g, etc.,<br />
parameters<br />
Goal: Improve parameters y and<br />
compute x<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
Outl<strong>in</strong>e<br />
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
1 The L<strong>in</strong>ear Problem: b = Ax + e<br />
2 The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
3 Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
4 Conclud<strong>in</strong>g Remarks<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
The L<strong>in</strong>ear Problem<br />
Assume A = A(y) is known exactly.<br />
We are given A and b, where<br />
b = Ax + e<br />
A is an ill-conditioned matrix, and we do not know e.<br />
We want to compute an approximation of x.<br />
Bad idea:<br />
e is small, so ignore it, and<br />
use x <strong>in</strong>v ≈ A −1 b<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
The L<strong>in</strong>ear Problem<br />
Assume A = A(y) is known exactly.<br />
We are given A and b, where<br />
b = Ax + e<br />
A is an ill-conditioned matrix, and we do not know e.<br />
We want to compute an approximation of x.<br />
Bad idea:<br />
e is small, so ignore it, and<br />
use x <strong>in</strong>v ≈ A −1 b<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Regularization Tools test problem: heat.m<br />
P. C. Hansen, www2.imm.dtu.dk/∼pch/Regutools<br />
Desired solution, x<br />
Noise free data, A*x<br />
1<br />
0.08<br />
0.07<br />
0.8<br />
0.06<br />
0.6<br />
0.05<br />
0.04<br />
0.4<br />
0.03<br />
0.2<br />
0.02<br />
0.01<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.01<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
If A and b are known exactly,<br />
can get an accurate reconstruction.<br />
Inverse solution x = A −1 b<br />
Noise free data, A*x<br />
1<br />
0.08<br />
0.07<br />
0.8<br />
0.06<br />
0.6<br />
0.05<br />
0.04<br />
0.4<br />
0.03<br />
0.2<br />
0.02<br />
0.01<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.01<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
But, if b conta<strong>in</strong>s a small amount of noise,<br />
Desired solution, x<br />
Noisy data, b = A*x + e<br />
1<br />
0.08<br />
0.07<br />
0.8<br />
0.06<br />
0.6<br />
0.05<br />
0.04<br />
0.4<br />
0.03<br />
0.2<br />
0.02<br />
0.01<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.01<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
But, if b conta<strong>in</strong>s a small amount of noise,<br />
then we get a poor reconstruction!<br />
Inverse solution x = A −1 b<br />
Noisy data, b = A*x + e<br />
1<br />
0.08<br />
0.07<br />
0.8<br />
0.06<br />
0.6<br />
0.05<br />
0.04<br />
0.4<br />
0.03<br />
0.2<br />
0.02<br />
0.01<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.01<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
SVD Analysis<br />
An important l<strong>in</strong>ear algebra tool: S<strong>in</strong>gular Value Decomposition<br />
Let A = UΣV T where<br />
Σ =diag(σ 1 , σ 2 , . . . , σ n ) , σ 1 ≥ σ 2 ≥ · · · ≥ σ n ≥ 0<br />
U T U = I , V T V = I<br />
U = [ u 1 u 2 · · · u n<br />
]<br />
V = [ v 1 v 2 · · · v n<br />
]<br />
(left s<strong>in</strong>gular vectors)<br />
(right s<strong>in</strong>gular vectors)<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
SVD Analysis<br />
The naïve <strong>in</strong>verse solution can then be represented as:<br />
x = A −1 b<br />
= VΣ −1 U T b<br />
=<br />
n∑<br />
i=1<br />
u T i<br />
b<br />
v i<br />
σ i<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
SVD Analysis<br />
The naïve <strong>in</strong>verse solution can then be represented as:<br />
ˆx = A −1 (b + e)<br />
= VΣ −1 U T (b + e)<br />
=<br />
n∑<br />
i=1<br />
u T i<br />
(b + e)<br />
v i<br />
σ i<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
SVD Analysis<br />
The naïve <strong>in</strong>verse solution can then be represented as:<br />
ˆx = A −1 (b + e)<br />
= VΣ −1 U T (b + e)<br />
=<br />
=<br />
n∑<br />
i=1<br />
n∑<br />
i=1<br />
u T i<br />
(b + e)<br />
v i<br />
σ i<br />
u T i<br />
b<br />
σ i<br />
v i +<br />
n∑<br />
i=1<br />
u T i<br />
e<br />
v i<br />
σ i<br />
= x + error<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
10 0 S<strong>in</strong>gular values<br />
10 −1<br />
10 −2<br />
10 −3<br />
10 −4<br />
10 −5<br />
10 −6<br />
10 −7<br />
10 −8<br />
10 −9<br />
10 −10<br />
0 50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Large σ i ↔ smooth (low frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 1<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Large σ i ↔ smooth (low frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 2<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Large σ i ↔ smooth (low frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 3<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Large σ i ↔ smooth (low frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 4<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Large σ i ↔ smooth (low frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 5<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 25<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 50<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 75<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 100<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 125<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Error term depends on s<strong>in</strong>gular values σ i and s<strong>in</strong>gular vectors v i .<br />
Small σ i ↔ oscillat<strong>in</strong>g (high frequency) v i<br />
10 0 S<strong>in</strong>gular values<br />
0.2<br />
S<strong>in</strong>gular vector, v 150<br />
10 −1<br />
10 −2<br />
0.15<br />
10 −3<br />
0.1<br />
10 −4<br />
0.05<br />
10 −5<br />
0<br />
10 −6<br />
−0.05<br />
10 −7<br />
10 −8<br />
−0.1<br />
10 −9<br />
−0.15<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
SVD Analysis<br />
The naïve <strong>in</strong>verse solution can then be represented as:<br />
ˆx = A −1 (b + e)<br />
= VΣ −1 U T (b + e)<br />
=<br />
=<br />
n∑<br />
i=1<br />
n∑<br />
i=1<br />
u T i<br />
(b + e)<br />
v i<br />
σ i<br />
u T i<br />
b<br />
σ i<br />
v i +<br />
n∑<br />
i=1<br />
u T i<br />
e<br />
v i<br />
σ i<br />
= x + error<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Regularization by Filter<strong>in</strong>g<br />
Basic Idea: Filter out effects of small s<strong>in</strong>gular values.<br />
(Hansen, SIAM, 1997)<br />
x reg = A −1<br />
regb = VΦΣ −1 U T b =<br />
where Φ = diag(φ 1 , φ 2 , . . . , φ n )<br />
n∑<br />
i=1<br />
φ i<br />
u T i<br />
b<br />
σ i<br />
v i ,<br />
The ”filter factors” satisfy<br />
{ 1 if σi is large<br />
φ i ≈<br />
0 if σ i is small<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
An Example: Tikhonov Regularization<br />
{<br />
m<strong>in</strong> ‖b − Ax‖<br />
2<br />
x<br />
2 + λ 2 ‖x‖ 2 }<br />
2<br />
⇔<br />
m<strong>in</strong><br />
x<br />
∥ ∥∥∥<br />
[ b<br />
0<br />
] [ A<br />
−<br />
λI<br />
]<br />
x<br />
∥<br />
2<br />
2<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
An Example: Tikhonov Regularization<br />
{<br />
m<strong>in</strong> ‖b − Ax‖<br />
2<br />
x<br />
2 + λ 2 ‖x‖ 2 }<br />
2<br />
⇔<br />
m<strong>in</strong><br />
x<br />
∥ ∥∥∥<br />
[ b<br />
0<br />
] [ A<br />
−<br />
λI<br />
]<br />
x<br />
∥<br />
2<br />
2<br />
An equivalent SVD filter<strong>in</strong>g formulation:<br />
x tik =<br />
n∑<br />
σi<br />
2 σ i<br />
i=1<br />
σ 2 i<br />
+ λ 2 u T i<br />
b<br />
v i<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
An Example: Tikhonov Regularization<br />
{<br />
m<strong>in</strong> ‖b − Ax‖<br />
2<br />
x<br />
2 + λ 2 ‖x‖ 2 }<br />
2<br />
⇔<br />
m<strong>in</strong><br />
x<br />
∥ ∥∥∥<br />
[ b<br />
0<br />
] [ A<br />
−<br />
λI<br />
]<br />
x<br />
∥<br />
2<br />
2<br />
An equivalent SVD filter<strong>in</strong>g formulation:<br />
1<br />
0.8<br />
x tik =<br />
n∑<br />
i=1<br />
σi<br />
2 u T i<br />
b<br />
σi 2 + λ 2<br />
filter factor<br />
0.6<br />
0.4<br />
0.2<br />
α = 0.001<br />
v i<br />
σ i<br />
0<br />
10 −5 10 −4 10 −3 10 −2 10 −1 10 0 10 1<br />
s<strong>in</strong>gular values<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Choos<strong>in</strong>g Regularization Parameters<br />
Lots of choices: Generalized Cross Validation (GCV), L-curve,<br />
discrepancy pr<strong>in</strong>ciple, ...<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Choos<strong>in</strong>g Regularization Parameters<br />
Lots of choices: Generalized Cross Validation (GCV), L-curve,<br />
discrepancy pr<strong>in</strong>ciple, ...<br />
GCV and Tikhonov: Choose λ to m<strong>in</strong>imize<br />
GCV(λ) =<br />
n<br />
n∑<br />
i=1<br />
( n∑<br />
i=1<br />
( u<br />
T<br />
i<br />
b<br />
σ 2 i<br />
+ λ 2 ) 2<br />
) 2<br />
1<br />
σi 2 + λ 2<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Reconstruction us<strong>in</strong>g Tikhonov reg. can be better than x <strong>in</strong>v .<br />
Quality of reconstruction depends on λ.<br />
But λ depends on A and b.<br />
Desired solution, x<br />
Inverse solution x = A −1 b<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Reconstruction us<strong>in</strong>g Tikhonov reg. can be better than x <strong>in</strong>v .<br />
Quality of reconstruction depends on λ.<br />
But λ depends on A and b.<br />
Regularized Solution, λ = 0.0005<br />
Inverse solution x = A −1 b<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Reconstruction us<strong>in</strong>g Tikhonov reg. can be better than x <strong>in</strong>v .<br />
Quality of reconstruction depends on λ.<br />
But λ depends on A and b.<br />
Regularized Solution, λ = 0.05<br />
Inverse solution x = A −1 b<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
Reconstruction us<strong>in</strong>g Tikhonov reg. can be better than x <strong>in</strong>v .<br />
Quality of reconstruction depends on λ.<br />
But λ depends on A and b.<br />
Regularized Solution, λ = 0.005<br />
Inverse solution x = A −1 b<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Filter<strong>in</strong>g for Large Scale <strong>Problems</strong><br />
Some remarks:<br />
For large matrices, comput<strong>in</strong>g SVD is expensive.<br />
SVD algorithms do not readily simplify for structured or<br />
sparse matrices.<br />
Alternative for large scale problems: LSQR iteration<br />
(Paige and Saunders, ACM TOMS, 1982)<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Lanczos Bidiagonalization (LBD)<br />
Given A and b, for k = 1, 2, ..., compute<br />
W k = [ ]<br />
w 1 w 2 · · · w k w k+1 , w1 = b/||b||<br />
Z k = [ ]<br />
z 1 z 2 · · · z k<br />
⎡<br />
⎤<br />
α 1<br />
β 2 α 2<br />
B k =<br />
. .. . ..<br />
⎢<br />
⎥<br />
⎣ β k α k<br />
⎦<br />
β k+1<br />
where W k and Z k have orthonormal columns, and<br />
A T W k = Z k B T k + α k+1z k+1 e T k+1<br />
AZ k = W k B k<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
LBD and LSQR<br />
At kth LBD iteration, use QR to solve projected LS problem:<br />
m<strong>in</strong> ‖b −<br />
x∈R(Z k ) Ax‖2 2 = m<strong>in</strong> ‖Wk T b − B kf‖ 2 2 = m<strong>in</strong> ‖βe 1 − B k f‖ 2 2<br />
f<br />
f<br />
where x k = Z k f<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
LBD and LSQR<br />
At kth LBD iteration, use QR to solve projected LS problem:<br />
m<strong>in</strong> ‖b −<br />
x∈R(Z k ) Ax‖2 2 = m<strong>in</strong> ‖Wk T b − B kf‖ 2 2 = m<strong>in</strong> ‖βe 1 − B k f‖ 2 2<br />
f<br />
f<br />
where x k = Z k f<br />
For our ill-posed <strong>in</strong>verse problems:<br />
S<strong>in</strong>gular values of B k converge to k largest s<strong>in</strong>g. values of A.<br />
Thus, x k is <strong>in</strong> a subspace that approximates a subspace<br />
spanned by the large s<strong>in</strong>gular components of A.<br />
For k < n, x k is a regularized solution.<br />
x n = x <strong>in</strong>v = A −1 b (bad approximation)<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
S<strong>in</strong>gular values of B k converge to large s<strong>in</strong>gular values of A.<br />
Thus, for early iterations k: f = B k \ W k b<br />
x k = Z k f<br />
is a regularized reconstruction.<br />
10 0 LBD iteration, k = 6<br />
svd(A)<br />
svd(B k<br />
)<br />
1<br />
iteration = 5<br />
10 −2<br />
0.8<br />
10 −4<br />
0.6<br />
0.4<br />
10 −6<br />
0.2<br />
10 −8<br />
0<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
S<strong>in</strong>gular values of B k converge to large s<strong>in</strong>gular values of A.<br />
Thus, for early iterations k: f = B k \ W k b<br />
x k = Z k f<br />
is a regularized reconstruction.<br />
10 0 LBD iteration, k = 16<br />
svd(A)<br />
svd(B k<br />
)<br />
1<br />
iteration = 15<br />
10 −2<br />
0.8<br />
10 −4<br />
0.6<br />
0.4<br />
10 −6<br />
0.2<br />
10 −8<br />
0<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
S<strong>in</strong>gular values of B k converge to large s<strong>in</strong>gular values of A.<br />
Thus, for later iterations k: f = B k \ W k b<br />
x k = Z k f<br />
is a noisy reconstruction.<br />
10 0 LBD iteration, k = 26<br />
svd(A)<br />
svd(B k<br />
)<br />
1<br />
iteration = 25<br />
10 −2<br />
0.8<br />
10 −4<br />
0.6<br />
0.4<br />
10 −6<br />
0.2<br />
10 −8<br />
0<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
S<strong>in</strong>gular values of B k converge to large s<strong>in</strong>gular values of A.<br />
Thus, for later iterations k: f = B k \ W k b<br />
x k = Z k f<br />
is a noisy reconstruction.<br />
10 0 LBD iteration, k = 36<br />
svd(A)<br />
svd(B k<br />
)<br />
1<br />
iteration = 35<br />
10 −2<br />
0.8<br />
10 −4<br />
0.6<br />
0.4<br />
10 −6<br />
0.2<br />
10 −8<br />
0<br />
10 −10<br />
0 50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Lanczos Based Hybrid Methods<br />
To avoid noisy reconstructions, embed regularization <strong>in</strong> LBD:<br />
O’Leary and Simmons, SISSC, 1981.<br />
Björck, BIT 1988.<br />
Björck, Grimme, and Van Dooren, BIT, 1994.<br />
Larsen, PhD Thesis, 1998.<br />
Hanke, BIT 2001.<br />
Kilmer and O’Leary, SIMAX, 2001.<br />
Kilmer, Hansen, Español, SISC 2007.<br />
Chung, N, O’Leary, ETNA 2007<br />
(HyBR Implementation)<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Regularize the Projected <strong>Least</strong> <strong>Squares</strong> Problem<br />
To stabilize convergence, regularize the projected problem:<br />
∥[ ∥∥∥ βe1<br />
m<strong>in</strong><br />
f 0<br />
] [<br />
Bk<br />
−<br />
λI<br />
Note: B k is very small compared to A, so<br />
]<br />
f<br />
∥<br />
Can use “expensive” methods to choose λ (e.g., GCV)<br />
Very little regularization is needed <strong>in</strong> early iterations.<br />
GCV tends to choose too large λ for bidiagonal system.<br />
Our remedy: Use a weighted GCV (Chung, N, O’Leary, 2007)<br />
Can also use WGCV <strong>in</strong>formation to estimate stopp<strong>in</strong>g iteration<br />
(approach similar to Björck, Grimme, and Van Dooren, BIT, 1994).<br />
2<br />
2<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
LSQR (no regularization)<br />
HyBR (Tikhonov regularization)<br />
[ ] ∖ [ ]<br />
Bk Wk b<br />
f = B k \ W k b f =<br />
λ k I 0<br />
x k = Z k f<br />
x k = Z k f<br />
iteration = 5<br />
iteration = 5<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
λ = 0.0115<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
LSQR (no regularization)<br />
HyBR (Tikhonov regularization)<br />
[ ] ∖ [ ]<br />
Bk Wk b<br />
f = B k \ W k b f =<br />
λ k I 0<br />
x k = Z k f<br />
x k = Z k f<br />
iteration = 15<br />
iteration = 15<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
λ = 0.0074<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
LSQR (no regularization)<br />
HyBR (Tikhonov regularization)<br />
[ ] ∖ [ ]<br />
Bk Wk b<br />
f = B k \ W k b f =<br />
λ k I 0<br />
x k = Z k f<br />
x k = Z k f<br />
iteration = 25<br />
iteration = 25<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
λ = 0.0050<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: Inverse Heat Equation<br />
LSQR (no regularization)<br />
HyBR (Tikhonov regularization)<br />
[ ] ∖ [ ]<br />
Bk Wk b<br />
f = B k \ W k b f =<br />
λ k I 0<br />
x k = Z k f<br />
x k = Z k f<br />
iteration = 35<br />
iteration = 35<br />
1<br />
1<br />
0.8<br />
0.8<br />
0.6<br />
0.6<br />
λ = 0.0042<br />
0.4<br />
0.4<br />
0.2<br />
0.2<br />
0<br />
0<br />
−0.2<br />
50 100 150 200 250<br />
−0.2<br />
50 100 150 200 250<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem<br />
We want to f<strong>in</strong>d x and y so that<br />
b = A(y)x + e<br />
With Tikhonov regularization, solve<br />
m<strong>in</strong><br />
x,y<br />
[ A(y)<br />
∥ λI<br />
] [ b<br />
x −<br />
0<br />
]∥ ∥∥∥<br />
2<br />
As with l<strong>in</strong>ear problem, choos<strong>in</strong>g a good regularization<br />
parameter λ is important.<br />
Problem is l<strong>in</strong>ear <strong>in</strong> x, nonl<strong>in</strong>ear <strong>in</strong> y.<br />
y ∈ R p , x ∈ R n , with p ≪ n.<br />
2<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
<strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong><br />
Variable Projection Method:<br />
Implicitly elim<strong>in</strong>ate l<strong>in</strong>ear term.<br />
Optimize over nonl<strong>in</strong>ear term.<br />
Some general references:<br />
Golub and Pereyra, SINUM 1973 (also IP 2003)<br />
Kaufman, BIT 1975<br />
Osborne, SINUM 1975 (also ETNA 2007)<br />
Ruhe and Wed<strong>in</strong>, SIREV, 1980<br />
How to apply to <strong>in</strong>verse problems?<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Variable Projection Method<br />
Instead of optimiz<strong>in</strong>g over both x and y:<br />
m<strong>in</strong><br />
x,y<br />
φ(x, y) = m<strong>in</strong><br />
x,y<br />
[ A(y)<br />
∥ λI<br />
] [ b<br />
x −<br />
0<br />
]∥ ∥∥∥<br />
2<br />
2<br />
Let x(y) be solution of<br />
∥[ ∥∥∥ A(y)<br />
m<strong>in</strong> φ(x, y) = m<strong>in</strong><br />
x<br />
x λI<br />
] [ b<br />
x −<br />
0<br />
and then m<strong>in</strong>imize the reduced cost functional:<br />
m<strong>in</strong><br />
y<br />
ψ(y) , ψ(y) = φ(x(y), y)<br />
]∥ ∥∥∥<br />
2<br />
2<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Gauss-Newton Algorithm<br />
choose <strong>in</strong>itial y 0<br />
for k = 0, 1, 2, . . .<br />
x k = arg m<strong>in</strong><br />
x<br />
∥ ∥∥∥<br />
[ A(yk )<br />
λ k I<br />
] [ b<br />
x −<br />
0<br />
]∥ ∥∥∥2<br />
r k = b − A(y k ) x k<br />
d k = arg m<strong>in</strong><br />
d<br />
‖J ψ d − r k ‖ 2<br />
end<br />
y k+1 = y k + d k<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Gauss-Newton Algorithm with HyBR<br />
And we use HyBR to solve the l<strong>in</strong>ear subproblem:<br />
choose <strong>in</strong>itial y 0<br />
for k = 0, 1, 2, . . .<br />
x k =HyBR(A(y k ), b)<br />
r k = b − A(y k ) x k<br />
d k = arg m<strong>in</strong><br />
d<br />
‖J ψ d − r k ‖ 2<br />
end<br />
y k+1 = y k + d k<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Matrix A(y) is def<strong>in</strong>ed by a PSF, which is <strong>in</strong> turn def<strong>in</strong>ed by<br />
parameters. Specifically:<br />
A(y) = A(P(y))<br />
where<br />
A is 65536 × 65536, with entries given by P.<br />
P is 256 × 256, with entries:<br />
( (i − k) 2 s2 2 p ij = exp<br />
− (j − l)2 s1 2 )<br />
+ 2(i − k)(j − l)ρ2<br />
2s1 2s2 2 − 2ρ4<br />
(k, l) is the PSF center (location of po<strong>in</strong>t source)<br />
y vector of unknown parameters:<br />
⎡ ⎤<br />
s 1<br />
y = ⎣ s 2<br />
ρ<br />
⎦<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Can get analytical formula for Jacobian:<br />
where x = vec(X).<br />
J ψ = ∂ { A( P(y) ) x }<br />
∂y<br />
= ∂<br />
∂<br />
{ A( P(y) ) x } ·<br />
∂P<br />
= A(X) ·<br />
∂<br />
∂y { P(y) }<br />
∂y { P(y) }<br />
Though <strong>in</strong> this example, f<strong>in</strong>ite difference approximation of J ψ<br />
works very well.<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Gauss-Newton Iteration History<br />
G-N Iteration ∆y λ<br />
0 0.5716 0.1685<br />
1 0.3345 0.1223<br />
2 0.2192 0.0985<br />
3 0.1473 0.0804<br />
4 0.1006 0.0715<br />
5 0.0648 0.0676<br />
6 0.0355 0.0657<br />
7 0.0144 0.0650<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Observed <strong>Image</strong> Reconstruction us<strong>in</strong>g <strong>in</strong>itial PSF Reconstruction after 8 GN iterations<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Conclud<strong>in</strong>g Remarks<br />
Imag<strong>in</strong>g applications require solv<strong>in</strong>g challeng<strong>in</strong>g <strong>in</strong>verse<br />
problems.<br />
<strong>Separable</strong> nonl<strong>in</strong>ear least squares models exploit high level<br />
structure.<br />
Hybrid methods are efficient solvers for large scale l<strong>in</strong>ear<br />
<strong>in</strong>verse problems.<br />
Automatic estimation of regularization parameter.<br />
Automatic estimation of stopp<strong>in</strong>g iteration.<br />
Hybrid methods can be effective l<strong>in</strong>ear solvers for nonl<strong>in</strong>ear<br />
problems.<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g
The L<strong>in</strong>ear Problem: b = Ax + e<br />
The <strong>Nonl<strong>in</strong>ear</strong> Problem: b = A(y) x + e<br />
Example: <strong>Image</strong> Deblurr<strong>in</strong>g<br />
Conclud<strong>in</strong>g Remarks<br />
Questions?<br />
Other methods to choose regularization parameters?<br />
Other regularization methods (e.g., total variation)?<br />
Sparse (<strong>in</strong> some basis) reconstructions?<br />
MATLAB Codes and Data?<br />
www.mathcs.emory.edu/∼nagy/WGCV<br />
www.mathcs.emory.edu/∼nagy/RestoreTools<br />
www2.imm.dtu.dk/∼pch/HNO<br />
www2.imm.dtu.dk/∼pch/Regutools<br />
Julianne Chung and James Nagy Emory University Atlanta, GA, USA <strong>Separable</strong> <strong>Nonl<strong>in</strong>ear</strong> <strong>Least</strong> <strong>Squares</strong> <strong>Problems</strong> <strong>in</strong> <strong>Image</strong> Process<strong>in</strong>g