sparse image representation via combined transforms - Convex ...

2.1. IMAGE CODING 7
2.1.2 Source and Channel Coding
After reviewing Shannon’s abstraction of communication systems, we focus our attention on
the transmitter. Usually, the decoding scheme at the receiver is determined by the encoding
scheme at the transmitter. There are two components in the transmitter (as shown in Figure
2.2): source coder and channel coder. For a discrete-time source, without loss of generality,
the role of a source coder is to code a message or a sequence of messages into as small a
number of bits as possible. The role of a channel coder is to transfer a bit stream into a
signal that is compatible with the channel and, at the same time, perform error correcting
to achieve an arbitrarily small probability of bit error.
TRANSMITTER
SOURCE
CODER
CHANNEL
CODER
MESSAGE
BITS
SIGNAL
Figure 2.2: Separation of encoding into source and channel coding.
Shannon’s Fundamental Theorem of communication has two parts: the direct part and
the converse part. The direct part states that if the minimum achievable source coding
rate of a given source is strictly below the capacity of a channel, then the source can be
reliably transmitted through the channel by appropriate encoding and decoding operations;
the converse part states that if the source coding rate is strictly greater than the capacity of
channel, then a reliable transmission is impossible. Shannon’s theorem implies that reliable
transmission can be separated into two operations: source coding and channel coding. The
design of source encoding and decoding can be independent of the characteristic of the
channel; similarly, the design of channel encoding and decoding can be independent of the
characteristics of the source. Owing to the converse theorem, the reliable transmission is
doable either through a combination of separated source and channel coding or not possible
at all—whether it is a joint source channel coding or not. Note that in Shannon’s proof, the
Fundamental Theorem requires the condition that both source and channel are stationary
and memoryless. A detailed discussion about whether the Fundamental Theorem holds in
more general scenarios is given in [138].
One fact I must point out is that in statistical language, Shannon’s theorem is an