University of Toronto
Department of Electrical & Computer Engineering

ECE 1502F, 2007

Information Theory

Instructor: F. R. Kschischang, BA4132
E-mail: frank@comm.utoronto.ca
Lectures: Tuesdays, 10:30 am - 12:00 pm, GB221
 Fridays, 10:30 - 12:00 pm, GB244
This course is an introduction to information theory. As stated by Cover and Thomas, the authors of the recommended text,
As many of the following topics as permitted by the time available will be covered.
Entropy:
entropy, relative entropy, mutual information, the asymptotic equipartition property, entropy rates for stochastic processes.
Data Compression:
the Kraft inequality, Shannon-Fano codes, Huffman codes, arithmetic coding.
Channel Capacity:
discrete channels, the random coding bound and its converse, Gaussian channels, coloured Gaussian noise and "water-filling."
Rate Distortion Theory:
the rate-distortion function, Hamming distortion, mean squared difference distortion, Gaussian sources.
Network Information Theory:
multiple user channels, broadcast channels.
Main Reference:
Thomas M. Cover and Joy A. Thomas, Elements of Information Theory, John Wiley & Sons, Inc., 1991.
Other References:
Robert G. Gallager, Information Theory and Reliable Communication, John Wiley & Sons, Inc., 1968.
Richard E. Blahut, Principles and Practice of Information Theory, Addison-Wesley, 1987.
Raymond W. Yeung, A First Course in Information Theory, Kluwer, 2002.
David J. C. MacKay, Information Theory, Inference, and Learning Algorithms, Cambridge University Press, 2003 and online.
IEEE Transactions on Information Theory, Commemorative Issue, Vol. 44, October 1998, online via IEEE Xplore.
Grading:
Final grades will be determined on the basis of problems sets and midterm and final examinations.
Links:
Handouts: Announcement: