sparse image representation via combined transforms - Convex ...

6 CHAPTER 2. SPARSITY IN IMAGE CODING
Another intuitive way to view this is that if p(x) were a uniform distribution over X, then
the entropy rate H(p(·)) would be log 2 |X|, where |X| is the cardinality (size) of the set X.
In (2.1) we can view 1/ log 2 p(x) as an analogue to the cardinality such that the entropy is
the average logarithm of it.
The channel capacity, similarly, is the average of the logarithm of the number of bits
(binary numbers from {0, 1}) that can be reliably distinguished at receiver. Let Y denote
the output of the channel. The channel capacity is defined as
C = max I(X, Y ),
p(X)
where I(X, Y ) is the mutual information between channel input X and channel output Y :
I(X, Y )=H(X)+H(Y ) − H(X, Y ).
[33] gives good examples in introducing the channel capacity concept.
The distortion is the statistical average of a (usually convex) function of the difference
between the estimation ˆX at the receiver and the original message X. Letd(ˆx − x) be the
measure of deviation. There are many options for function d: e.g., (1) d(ˆx − x) =(ˆx − x) 2 ,
(2) d(ˆx − x) =|ˆx − x|. The distortion D is simply the statistical mean of d(ˆx − x):
∫
D = d(ˆx − x)p(x)dx.
For (1), the distortion D is the residual mean square (RMS); for (2), the distortion D is
the mean of the absolute deviation (MAD).
The Fundamental Theorem for a discrete channel with noise [127] states that communication
with an arbitrarily small probability of error is possible (from an asymptotic point
of view, by coding many messages at once) if HC.
A more practical approach is to code with a prescribed maximum distortion D. This
is a topic that is covered by rate-distortion theory. The rate-distortion function gives the
minimum rate needed to approximate a source for a given distortion. A well-known book
on this topic is [10] by Berger.