sparse image representation via combined transforms - Convex ...

4.9. ITERATIVE METHODS 95
1. Avoiding introducing the dual variables as in the primal-dual log-barrier perturbed LP
approach. We transfer the problem to a unconstrained convex optimization problem
and apply Newton methods directly to this problem. It is not necessary to introduce
the dual variables and we do not need to increase the size of the primal variable x
by two (this was done in CDS approach). It is not necessary to consider the KKT
conditions. Although the core problem—as in (4.14)—is very similar to the core
problem—as in (4.13)—of CDS approach, the derivation is simpler and more direct.
2. Instead of using a standard conjugate gradients methods, we choose LSQR, which is
analytically equivalent to CG algorithm but is numerically more stable.
Chen, Donoho and Saunders [26] have successfully developed software to carry out
numerical experiments with problems having the size of thousands by tens of thousands.
We choose to implement a new approach in the hope that it simplifies the algorithm and
hopefully increase the numerical efficiency. A careful comparison of our new approach with
the one previously implemented will be an interesting future research topic. At the same
time, we choose LSQR instead of CG. (Chen et al. showed how to use LSQR when δ>0
in (4.13), but their experiments used CG.)
4.9 Iterative Methods
In our problem, we choose an iterative solver to solve the system in (4.12) because the
Hessian is a positive definite matrix with special structure to facilitate fast algorithms for
matrix-vector multiplication. As we know, when there is a fast (low complexity) algorithm
for matrix-vector multiplication and assuming the iterative solver takes a moderate number
of iterations to converge, an iterative solver is an ideal tool to solve a system of linear
equations [74, 78].
Here, we have a fast algorithm to multiply with the Hessian H(x) because, as in (4.8),
the second term is a diagonal matrix, and we know it is an O(N) algorithm to multiply
with a diagonal matrix. The Φ in our model is a combination of transforms having fast
algorithms (for example, 2-D wavelet transforms, 2-D DCT and edgelet-like transforms), so
we have fast algorithms to multiply with Φ and Φ T (which is simply the adjoint). Hence
we have a fast algorithm for multiplying with the Hessian.
Choosing the “correct” iterative method is a big topic. The next chapter is dedicated
to this topic. We choose a variation of the conjugate gradient method: LSQR [116, 117].