sparse image representation via combined transforms - Convex ...

2.2. SPARSITY AND COMPRESSION 13
Predictive Coding
Predictive coding is an alternative to transform coding. It is another popular technique
being used in source coding and quantization. The idea is to use a feedback loop to reduce
the redundancy (or correlation) in the signal. A review of the history and the recent
developments of the predictive coding in quantization can be found in [77].
Summary and Problem
For an image transmission system applying transform coding, the general scheme is depicted
in Figure 2.4. First of all, a transform is applied to the image, and a coefficient vector y is
obtained. Generally speaking, the sparser the vector y is, the easier it can be compressed.
The coefficient vector y is quantized, compressed, and transmitted to the other end of
the communication system. At the receiver, the observed vector ỹ should be a very good
approximation to the original coefficient vector y. An inverse transform is applied to recover
the image.
We have not answered the question on how to quantify the sparsity of a vector and why
the sparsity leads to a “good” compression. In the next section, to answer these questions,
we introduce the work of Donoho that gives a quantification of sparsity and its implication
in effective bit-level compression.
2.2 Sparsity and Compression
In the previous section, we imply a slogan: sparsity leads to efficient coding. In this section,
we first quantify the measure of sparsity, then explain an implication of sparsity to efficiency
of coding. Here coding means a cascade of a quantizer and a bit-level compresser. In the
transform coding, the following question is answered to some extent:
“What should the output of a transform be to simplify the subsequential quantization
and bit-level compression?”
There are two important approaches to quantify the difficulty (or the limitation) of
coding. One is a statistical approach, which is sometimes called an entropy approach. The
other is a deterministic approach introduced mainly by Donoho in [46] and [47]. We describe
both of them.
First, the statistical approach. If we know the probability density function, we can calculate
the entropy defined in (2.1). In fact, the entropy is a lower bound on the expected