Statistical Learning Under Communication Constraints

Abstract: Statistical learning theory is concerned with modeling statistical dependencies between an input random variable and an output random variable on the basis of training examples, with minimal prior knowledge of the underlying probability distribution. The basic assumption in learning theory is that the learning agent can observe the training data with arbitrary precision. This assumption, however, is not realistic in distributed settings, when the location where the training data are gathered is geographically separated from the site where the actual learning is done. In this talk, I will present some results on learning under communication rate constraints. In particular, I will focus on the problem of learning when the learning agent knows the input part of the training sequence precisely, but receives the output part over a noiseless digital channel of finite capacity. I will present bounds on the generalization error of learning algorithms operating under such communication constraints and illustrate the theory with some examples.

This work was presented in part at the 2007 IEEE Information Theory Workshop in Lake Tahoe, CA.

Maxim Raginsky received the B.S. and M.S. degrees in 2000 and the
Ph.D. degree in 2002 from Northwestern University, Evanston, IL,
all in Electrical Engineering. From 2002 to 2004 he was a Postdoctoral
Researcher at the Center for Photonic Communication and Computing at
Northwestern University, where he pursued work on quantum cryptography
and quantum communication and information theory. From 2004 to 2007 he
was a Beckman Foundation Postdoctoral Fellow at the University of
Illinois in Urbana-Champaign, where he carried out research on
information theory, statistical learning and computational
neuroscience. He has joined the Department of Electrical and Computer
Engineering at Duke University in the Fall of 2007 as a research
scientist. His interests include statistical signal processing,
information theory, statistical learning and nonparametric
estimation. He is particularly interested in problems that combine the
communication, signal processing and machine learning components in a
novel and nontrivial way, as well as in the theory and practice
of robust statistical inference with limited information.