Separable Nonlinear Least Squares Problems in Image Processing

The Linear Problem: b = Ax + e 

The Nonlinear Problem: b = A(y) x + e 

Example: Image Deblurring 

Concluding Remarks 

Separable Nonlinear Least Squares Problems 

in Image Processing 

Collaborators: 

Julianne Chung and James Nagy 

Emory University 

Atlanta, GA, USA 

Eldad Haber (Emory) 

Per Christian Hansen (Tech. Univ. of Denmark) 

Dianne O’Leary (University of Maryland) 

Julianne Chung and James Nagy Emory University Atlanta, GA, USA Separable Nonlinear Least Squares Problems in Image Processing





Inverse Problems in Imaging 

Imaging problems are often modeled as: 

b = Ax + e 

where 

A - large, ill-conditioned matrix 

b - known, measured (image) data 

e - noise, statistical properties may be known 

Goal: Compute approximation of image x 






Inverse Problems in Imaging 

A more realistic image formation model is: 

where 

b = A(y) x + e 

A(y) - large, ill-conditioned matrix 

b - known, measured (image) data 

e - noise, statistical properties may be known 

y - parameters defining A, usually approximated 

Goal: Compute approximation of image x 

and improve estimate of parameters y 






Application: Image Deblurring 

b = A(y) x + e = observed image 

where y describes blurring function 

Given: b and an estimate of y 

Standard Image Deblurring: 

Compute approximation of x 

Better approach: 

Jointly improve estimate of y 

and compute approximation of x. 

Observed Image 















Observed Image 















Reconstruction using initial PSF 















Reconstruction after 8 GN iterations 






Application: Image Data Fusion 

b j = A(y j ) x + e j 

(collected low resolution images) 

1−th low resolution image 







b j = A(y j ) x + e j 









b j = A(y j ) x + e j 









b j = A(y j ) x + e j 









b j = A(y j ) x + e j 









b j = A(y j ) x + e j 


⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 

b 1 A(y 1 ) e 1 

⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 

⎣ . ⎦ = ⎣ . ⎦ x+ ⎣ . ⎦ 

b m A(y m ) e m 

} {{ } } {{ } } {{ } 


b = A(y) x + e 

y = registration, blurring, etc., 

parameters 

Goal: Improve parameters y and 

compute x 







b j = A(y j ) x + e j 


⎡ ⎤ ⎡ ⎤ ⎡ ⎤ 

b 1 A(y 1 ) e 1 

⎢ ⎥ ⎢ ⎥ ⎢ ⎥ 

⎣ . ⎦ = ⎣ . ⎦ x+ ⎣ . ⎦ 

b m A(y m ) e m 

} {{ } } {{ } } {{ } 

Reconstructed high resolution image 

b = A(y) x + e 

y = registration, blurring, etc., 

parameters 

Goal: Improve parameters y and 

compute x 


Outline 





1 The Linear Problem: b = Ax + e 

2 The Nonlinear Problem: b = A(y) x + e 

3 Example: Image Deblurring 

4 Concluding Remarks 






The Linear Problem 

Assume A = A(y) is known exactly. 

We are given A and b, where 

b = Ax + e 

A is an ill-conditioned matrix, and we do not know e. 

We want to compute an approximation of x. 

Bad idea: 

e is small, so ignore it, and 

use x inv ≈ A −1 b 






The Linear Problem 

Assume A = A(y) is known exactly. 

We are given A and b, where 

b = Ax + e 

A is an ill-conditioned matrix, and we do not know e. 

We want to compute an approximation of x. 

Bad idea: 

e is small, so ignore it, and 

use x inv ≈ A −1 b 






Example: Inverse Heat Equation 

Regularization Tools test problem: heat.m 

P. C. Hansen, www2.imm.dtu.dk/∼pch/Regutools 

Desired solution, x 

Noise free data, A*x 

1 

0.08 

0.07 

0.8 

0.06 

0.6 

0.05 

0.04 

0.4 

0.03 

0.2 

0.02 

0.01 

0 

0 

−0.2 

50 100 150 200 250 

−0.01 

50 100 150 200 250 







If A and b are known exactly, 

can get an accurate reconstruction. 

Inverse solution x = A −1 b 

Noise free data, A*x 

1 

0.08 

0.07 

0.8 

0.06 

0.6 

0.05 

0.04 

0.4 

0.03 

0.2 

0.02 

0.01 

0 

0 

−0.2 

50 100 150 200 250 

−0.01 

50 100 150 200 250 







But, if b contains a small amount of noise, 


Noisy data, b = A*x + e 

1 

0.08 

0.07 

0.8 

0.06 

0.6 

0.05 

0.04 

0.4 

0.03 

0.2 

0.02 

0.01 

0 

0 

−0.2 

50 100 150 200 250 

−0.01 

50 100 150 200 250 







But, if b contains a small amount of noise, 

then we get a poor reconstruction! 


Noisy data, b = A*x + e 

1 

0.08 

0.07 

0.8 

0.06 

0.6 

0.05 

0.04 

0.4 

0.03 

0.2 

0.02 

0.01 

0 

0 

−0.2 

50 100 150 200 250 

−0.01 

50 100 150 200 250 






SVD Analysis 

An important linear algebra tool: Singular Value Decomposition 

Let A = UΣV T where 

Σ =diag(σ 1 , σ 2 , . . . , σ n ) , σ 1 ≥ σ 2 ≥ · · · ≥ σ n ≥ 0 

U T U = I , V T V = I 

U = [ u 1 u 2 · · · u n 

] 

V = [ v 1 v 2 · · · v n 

] 

(left singular vectors) 

(right singular vectors) 






SVD Analysis 

The naïve inverse solution can then be represented as: 

x = A −1 b 

= VΣ −1 U T b 

= 

n∑ 

i=1 

u T i 

b 

v i 

σ i 






SVD Analysis 


ˆx = A −1 (b + e) 

= VΣ −1 U T (b + e) 

= 

n∑ 

i=1 

u T i 

(b + e) 

v i 

σ i 






SVD Analysis 


ˆx = A −1 (b + e) 

= VΣ −1 U T (b + e) 

= 

= 

n∑ 

i=1 

n∑ 

i=1 

u T i 

(b + e) 

v i 

σ i 

u T i 

b 

σ i 

v i + 

n∑ 

i=1 

u T i 

e 

v i 

σ i 

= x + error 







Error term depends on singular values σ i and singular vectors v i . 

10 0 Singular values 

10 −1 

10 −2 

10 −3 

10 −4 

10 −5 

10 −6 

10 −7 

10 −8 

10 −9 

10 −10 

0 50 100 150 200 250 








Large σ i ↔ smooth (low frequency) v i 


0.2 

Singular vector, v 1 

10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 








Small σ i ↔ oscillating (high frequency) v i 


0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 










0.2 


10 −1 

10 −2 

0.15 

10 −3 

0.1 

10 −4 

0.05 

10 −5 

0 

10 −6 

−0.05 

10 −7 

10 −8 

−0.1 

10 −9 

−0.15 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 






SVD Analysis 


ˆx = A −1 (b + e) 

= VΣ −1 U T (b + e) 

= 

= 

n∑ 

i=1 

n∑ 

i=1 

u T i 

(b + e) 

v i 

σ i 

u T i 

b 

σ i 

v i + 

n∑ 

i=1 

u T i 

e 

v i 

σ i 

= x + error 






Regularization by Filtering 

Basic Idea: Filter out effects of small singular values. 

(Hansen, SIAM, 1997) 

x reg = A −1 

regb = VΦΣ −1 U T b = 

where Φ = diag(φ 1 , φ 2 , . . . , φ n ) 

n∑ 

i=1 

φ i 

u T i 

b 

σ i 

v i , 

The ”filter factors” satisfy 

{ 1 if σi is large 

φ i ≈ 

0 if σ i is small 






An Example: Tikhonov Regularization 

{ 

min ‖b − Ax‖ 

2 

x 

2 + λ 2 ‖x‖ 2 } 

2 

⇔ 

min 

x 

∥ ∥∥∥ 

[ b 

0 

] [ A 

− 

λI 

] 

x 

∥ 

2 

2 







{ 


2 

x 

2 + λ 2 ‖x‖ 2 } 

2 

⇔ 


x 

∥ ∥∥∥ 

[ b 

0 

] [ A 

− 

λI 

] 

x 

∥ 

2 

2 

An equivalent SVD filtering formulation: 

x tik = 

n∑ 

σi 

2 σ i 

i=1 

σ 2 i 

+ λ 2 u T i 

b 

v i 







{ 


2 

x 

2 + λ 2 ‖x‖ 2 } 

2 

⇔ 


x 

∥ ∥∥∥ 

[ b 

0 

] [ A 

− 

λI 

] 

x 

∥ 

2 

2 

An equivalent SVD filtering formulation: 

1 

0.8 

x tik = 

n∑ 

i=1 

σi 

2 u T i 

b 

σi 2 + λ 2 

filter factor 

0.6 

0.4 

0.2 

α = 0.001 

v i 

σ i 

0 

10 −5 10 −4 10 −3 10 −2 10 −1 10 0 10 1 

singular values 






Choosing Regularization Parameters 

Lots of choices: Generalized Cross Validation (GCV), L-curve, 

discrepancy principle, ... 






Choosing Regularization Parameters 

Lots of choices: Generalized Cross Validation (GCV), L-curve, 

discrepancy principle, ... 

GCV and Tikhonov: Choose λ to minimize 

GCV(λ) = 

n 

n∑ 

i=1 

( n∑ 

i=1 

( u 

T 

i 

b 

σ 2 i 

+ λ 2 ) 2 

) 2 

1 

σi 2 + λ 2 







Reconstruction using Tikhonov reg. can be better than x inv . 

Quality of reconstruction depends on λ. 

But λ depends on A and b. 



1 

1 

0.8 

0.8 

0.6 

0.6 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 










Regularized Solution, λ = 0.0005 


1 

1 

0.8 

0.8 

0.6 

0.6 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 












1 

1 

0.8 

0.8 

0.6 

0.6 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 












1 

1 

0.8 

0.8 

0.6 

0.6 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 






Filtering for Large Scale Problems 

Some remarks: 

For large matrices, computing SVD is expensive. 

SVD algorithms do not readily simplify for structured or 

sparse matrices. 

Alternative for large scale problems: LSQR iteration 

(Paige and Saunders, ACM TOMS, 1982) 






Lanczos Bidiagonalization (LBD) 

Given A and b, for k = 1, 2, ..., compute 

W k = [ ] 

w 1 w 2 · · · w k w k+1 , w1 = b/||b|| 

Z k = [ ] 

z 1 z 2 · · · z k 

⎡ 

⎤ 

α 1 

β 2 α 2 

B k = 

. .. . .. 

⎢ 

⎥ 

⎣ β k α k 

⎦ 

β k+1 

where W k and Z k have orthonormal columns, and 

A T W k = Z k B T k + α k+1z k+1 e T k+1 

AZ k = W k B k 






LBD and LSQR 

At kth LBD iteration, use QR to solve projected LS problem: 

min ‖b − 

x∈R(Z k ) Ax‖2 2 = min ‖Wk T b − B kf‖ 2 2 = min ‖βe 1 − B k f‖ 2 2 

f 

f 

where x k = Z k f 






LBD and LSQR 

At kth LBD iteration, use QR to solve projected LS problem: 

min ‖b − 

x∈R(Z k ) Ax‖2 2 = min ‖Wk T b − B kf‖ 2 2 = min ‖βe 1 − B k f‖ 2 2 

f 

f 

where x k = Z k f 

For our ill-posed inverse problems: 

Singular values of B k converge to k largest sing. values of A. 

Thus, x k is in a subspace that approximates a subspace 

spanned by the large singular components of A. 

For k < n, x k is a regularized solution. 

x n = x inv = A −1 b (bad approximation) 







Singular values of B k converge to large singular values of A. 

Thus, for early iterations k: f = B k \ W k b 

x k = Z k f 

is a regularized reconstruction. 

10 0 LBD iteration, k = 6 

svd(A) 

svd(B k 

) 

1 

iteration = 5 

10 −2 

0.8 

10 −4 

0.6 

0.4 

10 −6 

0.2 

10 −8 

0 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 








Thus, for early iterations k: f = B k \ W k b 

x k = Z k f 

is a regularized reconstruction. 


svd(A) 

svd(B k 

) 

1 

iteration = 15 

10 −2 

0.8 

10 −4 

0.6 

0.4 

10 −6 

0.2 

10 −8 

0 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 








Thus, for later iterations k: f = B k \ W k b 

x k = Z k f 

is a noisy reconstruction. 


svd(A) 

svd(B k 

) 

1 


10 −2 

0.8 

10 −4 

0.6 

0.4 

10 −6 

0.2 

10 −8 

0 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 








Thus, for later iterations k: f = B k \ W k b 

x k = Z k f 

is a noisy reconstruction. 


svd(A) 

svd(B k 

) 

1 


10 −2 

0.8 

10 −4 

0.6 

0.4 

10 −6 

0.2 

10 −8 

0 

10 −10 

0 50 100 150 200 250 

−0.2 

50 100 150 200 250 






Lanczos Based Hybrid Methods 

To avoid noisy reconstructions, embed regularization in LBD: 

O’Leary and Simmons, SISSC, 1981. 

Björck, BIT 1988. 

Björck, Grimme, and Van Dooren, BIT, 1994. 

Larsen, PhD Thesis, 1998. 

Hanke, BIT 2001. 

Kilmer and O’Leary, SIMAX, 2001. 

Kilmer, Hansen, Español, SISC 2007. 

Chung, N, O’Leary, ETNA 2007 

(HyBR Implementation) 






Regularize the Projected Least Squares Problem 

To stabilize convergence, regularize the projected problem: 

∥[ ∥∥∥ βe1 


f 0 

] [ 

Bk 

− 

λI 

Note: B k is very small compared to A, so 

] 

f 

∥ 

Can use “expensive” methods to choose λ (e.g., GCV) 

Very little regularization is needed in early iterations. 

GCV tends to choose too large λ for bidiagonal system. 

Our remedy: Use a weighted GCV (Chung, N, O’Leary, 2007) 

Can also use WGCV information to estimate stopping iteration 

(approach similar to Björck, Grimme, and Van Dooren, BIT, 1994). 

2 

2 







LSQR (no regularization) 

HyBR (Tikhonov regularization) 

[ ] ∖ [ ] 

Bk Wk b 

f = B k \ W k b f = 

λ k I 0 

x k = Z k f 

x k = Z k f 

iteration = 5 

iteration = 5 

1 

1 

0.8 

0.8 

0.6 

0.6 

λ = 0.0115 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 









[ ] ∖ [ ] 

Bk Wk b 

f = B k \ W k b f = 

λ k I 0 

x k = Z k f 

x k = Z k f 



1 

1 

0.8 

0.8 

0.6 

0.6 

λ = 0.0074 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 









[ ] ∖ [ ] 

Bk Wk b 

f = B k \ W k b f = 

λ k I 0 

x k = Z k f 

x k = Z k f 



1 

1 

0.8 

0.8 

0.6 

0.6 

λ = 0.0050 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 









[ ] ∖ [ ] 

Bk Wk b 

f = B k \ W k b f = 

λ k I 0 

x k = Z k f 

x k = Z k f 



1 

1 

0.8 

0.8 

0.6 

0.6 

λ = 0.0042 

0.4 

0.4 

0.2 

0.2 

0 

0 

−0.2 

50 100 150 200 250 

−0.2 

50 100 150 200 250 






The Nonlinear Problem 

We want to find x and y so that 

b = A(y)x + e 

With Tikhonov regularization, solve 


x,y 

[ A(y) 

∥ λI 

] [ b 

x − 

0 

]∥ ∥∥∥ 

2 

As with linear problem, choosing a good regularization 

parameter λ is important. 

Problem is linear in x, nonlinear in y. 

y ∈ R p , x ∈ R n , with p ≪ n. 

2 






Separable Nonlinear Least Squares 

Variable Projection Method: 

Implicitly eliminate linear term. 

Optimize over nonlinear term. 

Some general references: 

Golub and Pereyra, SINUM 1973 (also IP 2003) 

Kaufman, BIT 1975 

Osborne, SINUM 1975 (also ETNA 2007) 

Ruhe and Wedin, SIREV, 1980 

How to apply to inverse problems? 






Variable Projection Method 

Instead of optimizing over both x and y: 


x,y 

φ(x, y) = min 

x,y 

[ A(y) 

∥ λI 

] [ b 

x − 

0 

]∥ ∥∥∥ 

2 

2 

Let x(y) be solution of 

∥[ ∥∥∥ A(y) 

min φ(x, y) = min 

x 

x λI 

] [ b 

x − 

0 

and then minimize the reduced cost functional: 


y 

ψ(y) , ψ(y) = φ(x(y), y) 

]∥ ∥∥∥ 

2 

2 






Gauss-Newton Algorithm 

choose initial y 0 

for k = 0, 1, 2, . . . 

x k = arg min 

x 

∥ ∥∥∥ 

[ A(yk ) 

λ k I 

] [ b 

x − 

0 

]∥ ∥∥∥2 

r k = b − A(y k ) x k 

d k = arg min 

d 

‖J ψ d − r k ‖ 2 

end 

y k+1 = y k + d k 






Gauss-Newton Algorithm with HyBR 

And we use HyBR to solve the linear subproblem: 

choose initial y 0 

for k = 0, 1, 2, . . . 

x k =HyBR(A(y k ), b) 

r k = b − A(y k ) x k 

d k = arg min 

d 

‖J ψ d − r k ‖ 2 

end 

y k+1 = y k + d k 







Matrix A(y) is defined by a PSF, which is in turn defined by 

parameters. Specifically: 

A(y) = A(P(y)) 

where 

A is 65536 × 65536, with entries given by P. 

P is 256 × 256, with entries: 

( (i − k) 2 s2 2 p ij = exp 

− (j − l)2 s1 2 ) 

+ 2(i − k)(j − l)ρ2 

2s1 2s2 2 − 2ρ4 

(k, l) is the PSF center (location of point source) 

y vector of unknown parameters: 

⎡ ⎤ 

s 1 

y = ⎣ s 2 

ρ 

⎦ 







Can get analytical formula for Jacobian: 

where x = vec(X). 

J ψ = ∂ { A( P(y) ) x } 

∂y 

= ∂ 

∂ 

{ A( P(y) ) x } · 

∂P 

= A(X) · 

∂ 

∂y { P(y) } 

∂y { P(y) } 

Though in this example, finite difference approximation of J ψ 

works very well. 







Gauss-Newton Iteration History 

G-N Iteration ∆y λ 

0 0.5716 0.1685 

1 0.3345 0.1223 

2 0.2192 0.0985 

3 0.1473 0.0804 

4 0.1006 0.0715 

5 0.0648 0.0676 

6 0.0355 0.0657 

7 0.0144 0.0650 







Observed Image Reconstruction using initial PSF Reconstruction after 8 GN iterations 







Imaging applications require solving challenging inverse 

problems. 

Separable nonlinear least squares models exploit high level 

structure. 

Hybrid methods are efficient solvers for large scale linear 

inverse problems. 

Automatic estimation of regularization parameter. 

Automatic estimation of stopping iteration. 

Hybrid methods can be effective linear solvers for nonlinear 

problems. 






Questions? 

Other methods to choose regularization parameters? 

Other regularization methods (e.g., total variation)? 

Sparse (in some basis) reconstructions? 

MATLAB Codes and Data? 

www.mathcs.emory.edu/∼nagy/WGCV 

www.mathcs.emory.edu/∼nagy/RestoreTools 

www2.imm.dtu.dk/∼pch/HNO 

www2.imm.dtu.dk/∼pch/Regutools

Separable Nonlinear Least Squares Problems in Image Processing

You also want an ePaper? Increase the reach of your titles

Delete template?

Save as template?