sparse image representation via combined transforms - Convex ...

96 CHAPTER 4. COMBINED IMAGE REPRESENTATION
4.10 Numerical Issues
Damped Newton direction. To ensure that Newton’s method converges, we implement a
backtracking scheme to find the value for β i in (4.11). It guarantees that at every iteration,
the value of the objective function is reduced.
Truncated CG. In order to save computing time, in the early Newton iterations, we terminate
the CG solver before it reaches high precision, because an inexact Newton direction
does not hurt the precision of the final solution by Newton method [38, 39].
The algorithm is implemented in Matlab. Fast algorithms for wavelet and edgelet transforms
are implemented in C and called by Matlab through a CMEX interface.
4.11 Discussion
4.11.1 Connection With Statistics
In statistics, we can find the same method being used in model selection, where we choose
a subset of variables so that the model is still sufficient for prediction and inference. To be
more specific, in linear regression models, we consider
y = Xβ + ε,
where y is the response, X is the model matrix with every column being values of a variable
(predictor), β is the coefficient vector, and ε is a vector of IID random variables. Model
selection in this setting means choosing a subset of columns of X, X (0) , so that for most
of the possible responses y, wehavey ≈ X (0) β (0) , where β (0) is a subvector of β with
locations corresponding to the selected columns in X (0) . The difference (or prediction
error), y − X (0) β (0) , is negligible in the sense that it can be interpreted as a realization of
the random noise vector ε.
Typically, people use penalized regression to select the model. Basically, we solve
(PR)
minimize ‖y − Xβ‖ 2 2 + λρ(β),
β
which is exactly the problem we encountered in (4.2). After solving problem (PR), we can
pick the ith column in X if β i has a significant amplitude. When ρ(β) =‖β‖ 1 , the method
(PR) is called LASSO by R. Tibshirani [133] and Basis Pursuit by Chen et al [27]. When