Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder

Zhengping Ji, Wentao Huang, Steven P. Brumby

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

Original languageEnglish (US)
Title of host publication2012 International Joint Conference on Neural Networks, IJCNN 2012
DOIs
StatePublished - 2012
Event2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012 - Brisbane, QLD, Australia
Duration: Jun 10 2012Jun 15 2012

Publication series

NameProceedings of the International Joint Conference on Neural Networks

Other

Other2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012
CountryAustralia
CityBrisbane, QLD
Period6/10/126/15/12

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder'. Together they form a unique fingerprint.

Cite this