Nonlinear system modeling with random matrices: Echo state networks revisited

Bai Zhang, David J. Miller, Yue Wang

Research output: Contribution to journalArticle

Abstract

Echo state networks (ESNs) are a novel form of recurrent neural networks (RNNs) that provide an efficient and powerful computational model approximating nonlinear dynamical systems. A unique feature of an ESN is that a large number of neurons (the 'reservoir') are used, whose synaptic connections are generated randomly, with only the connections from the reservoir to the output modified by learning. Why a large randomly generated fixed RNN gives such excellent performance in approximating nonlinear systems is still not well understood. In this brief, we apply random matrix theory to examine the properties of random reservoirs in ESNs under different topologies (sparse or fully connected) and connection weights (Bernoulli or Gaussian). We quantify the asymptotic gap between the scaling factor bounds for the necessary and sufficient conditions previously proposed for the echo state property. We then show that the state transition mapping is contractive with high probability when only the necessary condition is satisfied, which corroborates and thus analytically explains the observation that in practice one obtains echo states when the spectral radius of the reservoir weight matrix is smaller than 1.

Original languageEnglish (US)
Article number6105577
Pages (from-to)175-182
Number of pages8
JournalIEEE Transactions on Neural Networks and Learning Systems
Volume23
Issue number1
DOIs
StatePublished - 2012

Keywords

  • Circular law
  • concentration of measure
  • echo state networks
  • echo state property
  • random matrix theory
  • recurrent neural networks

ASJC Scopus subject areas

  • Software
  • Computer Science Applications
  • Computer Networks and Communications
  • Artificial Intelligence

Fingerprint Dive into the research topics of 'Nonlinear system modeling with random matrices: Echo state networks revisited'. Together they form a unique fingerprint.

  • Cite this