Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder

Zhengping Ji, Wentao Huang, Steven P. Brumby

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

Original languageEnglish (US)
Title of host publication2012 International Joint Conference on Neural Networks, IJCNN 2012
DOIs
StatePublished - Aug 22 2012
Event2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012 - Brisbane, QLD, Australia
Duration: Jun 10 2012Jun 15 2012

Other

Other2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012
CountryAustralia
CityBrisbane, QLD
Period6/10/126/15/12

Fingerprint

Gabor filters
Glossaries
Image reconstruction
Learning algorithms
Restoration
Decoding
Classifiers

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence

Cite this

Ji, Z., Huang, W., & Brumby, S. P. (2012). Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder. In 2012 International Joint Conference on Neural Networks, IJCNN 2012 [6252810] https://doi.org/10.1109/IJCNN.2012.6252810

Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder. / Ji, Zhengping; Huang, Wentao; Brumby, Steven P.

2012 International Joint Conference on Neural Networks, IJCNN 2012. 2012. 6252810.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Ji, Z, Huang, W & Brumby, SP 2012, Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder. in 2012 International Joint Conference on Neural Networks, IJCNN 2012., 6252810, 2012 Annual International Joint Conference on Neural Networks, IJCNN 2012, Part of the 2012 IEEE World Congress on Computational Intelligence, WCCI 2012, Brisbane, QLD, Australia, 6/10/12. https://doi.org/10.1109/IJCNN.2012.6252810
Ji Z, Huang W, Brumby SP. Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder. In 2012 International Joint Conference on Neural Networks, IJCNN 2012. 2012. 6252810 https://doi.org/10.1109/IJCNN.2012.6252810
Ji, Zhengping ; Huang, Wentao ; Brumby, Steven P. / Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder. 2012 International Joint Conference on Neural Networks, IJCNN 2012. 2012.
@inproceedings{2fb98eb0bb294acebe5f555f6ecbb260,
title = "Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder",
abstract = "Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.",
author = "Zhengping Ji and Wentao Huang and Brumby, {Steven P.}",
year = "2012",
month = "8",
day = "22",
doi = "10.1109/IJCNN.2012.6252810",
language = "English (US)",
isbn = "9781467314909",
booktitle = "2012 International Joint Conference on Neural Networks, IJCNN 2012",

}

TY - GEN

T1 - Learning sparse representation via a nonlinear shrinkage encoder and a linear sparse decoder

AU - Ji, Zhengping

AU - Huang, Wentao

AU - Brumby, Steven P.

PY - 2012/8/22

Y1 - 2012/8/22

N2 - Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

AB - Learning sparse representations for deep networks has drawn considerable research interest in recent years. In this paper, we present a novel framework to learn sparse representations via a generalized encoder-decoder architecture. The basic idea is to adopt a fast approximation to the iterative sparse coding solution and form an efficient nonlinear encoder to map an input to a sparse representation. A set of basis functions is then learned through the minimization of an energy function consisting of a sparseness prior and linear decoder constraints. Applying a greedy layer-wise learning scheme, this framework can be extended to more layers to learn deep networks. The proposed learning algorithm is also highly efficient as no iterative operations are required, and both batch and on-line learning are supported. Given the sparse representation and basis functions, an optimized decoding procedure is carried out to reconstruct and denoise the input signals. We evaluate our model on natural image patches to develop a dictionary of V1-like Gabor filters, and further show that basis functions in a higher layer (e.g., V2) combine the filters in a lower layer to generate more complex patterns to benefit the high-level tasks. We then use the sparse representations to recognize objects in two benchmark data sets (i.e., CIFAR-10 and NORB) via a linear SVM classifier, and demonstrate better or comparable recognition performances with respect to state-of-art algorithms. The image reconstruction of MNIST images and the restoration of corrupted versions are presented at the end.

UR - http://www.scopus.com/inward/record.url?scp=84865085847&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84865085847&partnerID=8YFLogxK

U2 - 10.1109/IJCNN.2012.6252810

DO - 10.1109/IJCNN.2012.6252810

M3 - Conference contribution

AN - SCOPUS:84865085847

SN - 9781467314909

BT - 2012 International Joint Conference on Neural Networks, IJCNN 2012

ER -