sparse image representation via combined transforms - Convex ...

2.2. SPARSITY AND COMPRESSION 15
average length of a bit string that is necessary to code a signal in a quantization-compression
scheme. A variety of schemes have been developed in the literature. In different circumstances,
some schemes approach this entropy lower bound. The field that includes this
research is called vector quantization.
Furthermore, if the coefficients after a certain transform are independent and identically
distributed (IID), then we can deploy the same scalar quantizer for every coordinate. The
scalar quantizer can nearly achieve the informatic lower bound (the lower bound specified
in information theory). For example, one can use the Lloyd-Max quantizer.
The statistical (or entropic) approach is elegant, but a practical disadvantage is that in
image coding we generally do not know the statistical distribution of an image. Images are
from an extremely high-dimensional space. Suppose a digital image has 256×256 pixels and
each pixel is assigned a real value. The dimensionality of the image is then 65, 536. Viewing
every image as a sample from a high-dimensional space, the number of samples is far less
than the dimensionality of the space. Think about how many images we can find on the
internet. The size of image databases is usually not beyond three digits; for example, the
USC image database and the database at the Center for Imaging Science Sensor. It is hard
to infer the probability distribution in such a high-dimensional space from such a small
sample. This motivates us to find a non-stochastic approach, which is also an empirical
approach. By “empirical” we mean that the method depends only on observations.
The key in the non-stochastic approach is to measure the empirical sparsity. We focus
on the ideas that have been developed in [46, 47]. The measure is called the weak l p norm.
The following two subsections answer the following two questions:
1. What is the weak l p norm and what are the scientific reasons to invent such a norm?
(Answered in Section 2.2.1.)
2. How is the weak l p norm related to asymptotic efficiency of coding? (Answered in
Section 2.2.2.)
2.2.1 Weak l p Norm
We start with the definition of the weak l p norm, for 1 ≤ p ≤ 2, then describe its connection
with compression number, measure of numerosity and rate of recovery, and finally we
introduce the concept of critical index, which plays an important role in the next section
(Section 2.2.2).