Information and Coding Theory

``Given two people of approximately the same ability and one person who works ten percent more than the other, the latter will more than twice out produce the former. The more you know, the more you learn; the more you learn, the more you can do; the more you can do, the more the opportunity - it is very much like compound interest. I don't want to give you a rate, but it is a very high rate. Given two people with exactly the same ability, the one person who manages day in and day out to get in one more hour of thinking will be tremendously more productive over a lifetime.......''

Richard Hamming - ``You and Your Research''

Information Theory developed as a discipline of its own after Shannon published his landmark paper "A Mathematical Theory of Communication" in the late 40s. Such was the genius of this man and his insight into the representation and formulation of communications in mathematical terms, that the paper published under the above title is also known as "The Mathematical Theory of Communication".

Information theory tries to answer the question of how to represent messages or symbols in a very short string of bits. For example: If we have 16 messages that are possible equally likely (p_k=1/16) then in binary communications we will require 4 bits to represent any of the messages.

What if the messages are not equally likely? Then one brute force approach (Huffman Coding) is to assign more bits to less likely messages and less number of bits to more likely messages.
The English alphabets (A-Z) do not appear in a equally likely way in a given text. For example in the above paragraph the letter 'e' appeared 18 times (more than any letter) . Thus it does make sense to encode 'e' in less number of bits if we would like transmit the above paragraph.

Thus assigning long symbols for less likely messages and short symbols for more likely messages is economical.
 

Favorite Books On the Topic

1. "
Elements of Information Theory "
Thomas M. Cover , Joy A. Thomas
2. 
   
" Information Theory and Reliable Communication "
Robert G. Gallager
3. 
   
"Mathematical Theory of Communication "
Claude E. Shannon, Warren Weaver

Standard bibliography On Information and Coding Theory

Shannon's original papers first appeared in book form in CLAUDE E. SHANNON and WARREN WEAVER, The Mathematical Theory of Communication (1949, reprinted 1975). The major papers in the development of information theory from 1948 to 1972, including Shannon's original work, are collected in DAVID SLEPIAN (ed.), Key Papers in the Development of Information Theory (1974).

Other early influential works are:

NORBERT WIENER: Extrapolation, Interpolation, and Smoothing of Stationary Time Series: With Engineering Applications (1949, reprinted 1966);
V.A. KOTELNIKOV: The Theory of Optimum Noise Immunity, trans. from Russian (1959);
ALEKSANDR KHINCHIN: Mathematical Foundations of Information Theory, trans. from Russian (1957);
AMIEL FEINSTEIN: Foundations of Information Theory (1958);
ROBERT M. FANO: Transmission of Information: A Statistical Theory of Communications (1961);
JACOB WOLFOWITZ: Coding Theorems of Information Theory, 3rd ed. (1978); and
M.S. PINSKER: Information and Information Stability of Random Variables and Processes, trans. from Russian (1964).

Elementary texts on information theory include

RICHARD W. HAMMING: Coding and Information Theory, 2nd ed. (1986);
NORMAN ABRAMSON: Information Theory and Coding (1963);
MASUD MANSURIPUR: Introduction to Information Theory (1987); and
JOHN R. PIERCE: An Introduction to Information Theory: Symbols, Signals & Noise, 2nd rev. ed. (1980).

More advanced and comprehensive general works include:

ROBERT G. GALLAGER: Information Theory and Reliable Communication (1968);
ROBERT J. McELIECE: The Theory of Information and Coding (1977, reissued 1984);
IMRE CSISZÁR and JÁNOS KÖRNER: Information Theory: Coding Theorems for Discrete Memoryless Systems (1981);
and ANDREW J. VITERBI and JIM K. OMURA: Principles of Digital Communication and Coding (1979).

Coding and decoding for error correction are treated in the above-mentioned general texts and in greater depth in :

RICHARD E. BLAHUT: Theory and Practice of Error Control Codes (1983);
SHU LIN and DANIEL J. COSTELLO, JR.: Error Control Coding: Fundamentals and Applications (1983);
ELWYN R. BERLEKAMP: Algebraic Coding Theory, rev. ed. (1984);
GEORGE C. CLARK, JR., and J. BIBB CAIN: Error-Correction Coding for Digital Communications (1981); and
W. WESLEY PETERSON and E.J. WELDON, JR.: Error-Correcting Codes, 2nd ed. (1972).

Sources with a distortion measure are covered in :

TOBY BERGER: Rate Distortion Theory: A Mathematical Basis for Data Compression (1971);
TOBY BERGER and LEE D. DAVISSON: Advances in Source Coding (1975); and in the above-mentioned general texts.

Treatments of detection, estimation, and prediction can be found in :

HARRY L. VAN TREES: Detection, Estimation, and Modulation Theory, 3 vol. (1968-71);
JAMES L. MELSA and DAVID L. COHN: Decision and Estimation Theory (1978);
and HARRY STARK and JOHN W. WOODS: Probability, Random Processes, and Estimation Theory for Engineers (1986).

The broader aspects of information theory and closely related fields are well presented in:

FRITZ MACHLUP and UNA MANSFIELD: The Study of Information: Interdisciplinary Messages (1983).

Other works applying information theory to various fields include:

JACOB MARSCHAK: Economic Information, Decision, and Prediction, 3 vol. (1974);
WENDELL R. GARNER: The Processing of Information and Structure (1974);
WERNER HOLZMÜLLER: Information in Biological Systems (1984 originally published in German, 1981);
SERAFIN FRAGA, K.M.S. SAXENA, and MANUEL TORRES: Biomolecular Information Theory (1978); and
KAREL ECKSCHLAGER and VLADIMÍRSTEPÁNEK: Information Theory as Applied to Chemical Analysis (1979), and Analytical Measurement and Information: Advances in the Information Theoretic Approach to Chemical Analyses (1985).