Learning Efficient Codes for Natural Images

Part I. Michael S. Lewicki

collaborator: Terrence J. Sejnowski

We derive a learning algorithm for inferring an overcomplete basis, ie more basis functions than input variables, by viewing it as probabilistic model of the observed data. Redundancy in the representation is removed by the prior on the basis coefficients which assigns probabilities to alternative representations. A Laplacian prior leads to representations that are sparse and are a nonlinear function of the data. In contrast to traditional complete bases, such as Fourier or Wavelet, the vectors in an overcomplete basis can become specialized for a larger variety of features present in the entire ensemble of data. This framework provides a new view of overcomplete bases, placing emphasis on their greater flexibility in modeling the underlying statistical density of the data, which allows for better noise reduction properties and greater coding efficiency.

Part II. Bruno A. Olshausen

We apply a general technique for learning overcomplete bases (Lewicki & Sejnowski, 1997) to the problem of finding efficient image codes. The bases learned by the algorithm are localized, oriented, and bandpass, consistent with earlier results obtained using different methods Olshausen and Field (1996); Bell and Sejnowski (1996). We show that higher degrees of overcompleteness produce bases which have much greater likelihood and results in a Gabor-like basis with greater sampling density in position, orientation, and scale. This framework also allows different bases to be compared objectively by calculating their probability given the observed data. Compared to the complete and overcomplete Fourier and wavelet bases, the learned bases have much greater probability and thus have the potential to yield better coding efficiency. We demonstrate the improvement in the representation of the learned bases by showing superior noise reduction properties.

Back to the Natural Scenes Meeting Agenda.