Parameter estimation in a highly non-linear model using simultaneous perturbation stochastic approximation

James E. Whitney, Kerron Duncan, Maria Richardson, Isaac Bankman

Research output: Contribution to journalArticle

Abstract

Often it is necessary to estimate the parameters of a model or unknown system. Various techniques exist to accomplish this task, including Kalman and Wiener filtering, least-mean-square (LMS) algorithms, and the Levenberg-Marquardt(L-M) algorithm. These techniques require an analytic form of the gradient of the function of the parameters to be estimated. A key feature of the simultaneous perturbation stochastic approximation (SPSA) method is that it is a gradient-free optimization technique (Spall; 1992,1998a,b, 1999). In the current problem, the function of parameters to be identified is highly non-linear and of sufficient difficulty that obtaining an analytic form of the gradient is impractical. Therefore, in this paper the performance of the SPSA algorithm will be examined in terms of parameter selection, data requirements, and convergence performance on this non-linear problem. Results will be reported on both a first-order "standard" implementation of SPSA and on a second-order version of SPSA that tends to enhance convergence.

Original languageEnglish (US)
Pages (from-to)1247-1256
Number of pages10
JournalCommunications in Statistics - Theory and Methods
Volume29
Issue number5-6
StatePublished - 2000

Fingerprint

Stochastic Approximation
Parameter estimation
Nonlinear Model
Parameter Estimation
Perturbation
Approximation algorithms
Gradient
Wiener Filtering
Levenberg-Marquardt Algorithm
Least Mean Square
Kalman Filtering
Parameter Selection
Stochastic Algorithms
Stochastic Methods
Approximation Methods
Optimization Techniques
Nonlinear Problem
Approximation Algorithms
Tend
Sufficient

Keywords

  • Optimization
  • SPSA

ASJC Scopus subject areas

  • Safety, Risk, Reliability and Quality
  • Statistics and Probability

Cite this

Parameter estimation in a highly non-linear model using simultaneous perturbation stochastic approximation. / Whitney, James E.; Duncan, Kerron; Richardson, Maria; Bankman, Isaac.

In: Communications in Statistics - Theory and Methods, Vol. 29, No. 5-6, 2000, p. 1247-1256.

Research output: Contribution to journalArticle

Whitney, James E. ; Duncan, Kerron ; Richardson, Maria ; Bankman, Isaac. / Parameter estimation in a highly non-linear model using simultaneous perturbation stochastic approximation. In: Communications in Statistics - Theory and Methods. 2000 ; Vol. 29, No. 5-6. pp. 1247-1256.
@article{b679cd2a933342c5b94cf3506bd148c8,
title = "Parameter estimation in a highly non-linear model using simultaneous perturbation stochastic approximation",
abstract = "Often it is necessary to estimate the parameters of a model or unknown system. Various techniques exist to accomplish this task, including Kalman and Wiener filtering, least-mean-square (LMS) algorithms, and the Levenberg-Marquardt(L-M) algorithm. These techniques require an analytic form of the gradient of the function of the parameters to be estimated. A key feature of the simultaneous perturbation stochastic approximation (SPSA) method is that it is a gradient-free optimization technique (Spall; 1992,1998a,b, 1999). In the current problem, the function of parameters to be identified is highly non-linear and of sufficient difficulty that obtaining an analytic form of the gradient is impractical. Therefore, in this paper the performance of the SPSA algorithm will be examined in terms of parameter selection, data requirements, and convergence performance on this non-linear problem. Results will be reported on both a first-order {"}standard{"} implementation of SPSA and on a second-order version of SPSA that tends to enhance convergence.",
keywords = "Optimization, SPSA",
author = "Whitney, {James E.} and Kerron Duncan and Maria Richardson and Isaac Bankman",
year = "2000",
language = "English (US)",
volume = "29",
pages = "1247--1256",
journal = "Communications in Statistics - Theory and Methods",
issn = "0361-0926",
publisher = "Taylor and Francis Ltd.",
number = "5-6",

}

TY - JOUR

T1 - Parameter estimation in a highly non-linear model using simultaneous perturbation stochastic approximation

AU - Whitney, James E.

AU - Duncan, Kerron

AU - Richardson, Maria

AU - Bankman, Isaac

PY - 2000

Y1 - 2000

N2 - Often it is necessary to estimate the parameters of a model or unknown system. Various techniques exist to accomplish this task, including Kalman and Wiener filtering, least-mean-square (LMS) algorithms, and the Levenberg-Marquardt(L-M) algorithm. These techniques require an analytic form of the gradient of the function of the parameters to be estimated. A key feature of the simultaneous perturbation stochastic approximation (SPSA) method is that it is a gradient-free optimization technique (Spall; 1992,1998a,b, 1999). In the current problem, the function of parameters to be identified is highly non-linear and of sufficient difficulty that obtaining an analytic form of the gradient is impractical. Therefore, in this paper the performance of the SPSA algorithm will be examined in terms of parameter selection, data requirements, and convergence performance on this non-linear problem. Results will be reported on both a first-order "standard" implementation of SPSA and on a second-order version of SPSA that tends to enhance convergence.

AB - Often it is necessary to estimate the parameters of a model or unknown system. Various techniques exist to accomplish this task, including Kalman and Wiener filtering, least-mean-square (LMS) algorithms, and the Levenberg-Marquardt(L-M) algorithm. These techniques require an analytic form of the gradient of the function of the parameters to be estimated. A key feature of the simultaneous perturbation stochastic approximation (SPSA) method is that it is a gradient-free optimization technique (Spall; 1992,1998a,b, 1999). In the current problem, the function of parameters to be identified is highly non-linear and of sufficient difficulty that obtaining an analytic form of the gradient is impractical. Therefore, in this paper the performance of the SPSA algorithm will be examined in terms of parameter selection, data requirements, and convergence performance on this non-linear problem. Results will be reported on both a first-order "standard" implementation of SPSA and on a second-order version of SPSA that tends to enhance convergence.

KW - Optimization

KW - SPSA

UR - http://www.scopus.com/inward/record.url?scp=18844369048&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=18844369048&partnerID=8YFLogxK

M3 - Article

AN - SCOPUS:18844369048

VL - 29

SP - 1247

EP - 1256

JO - Communications in Statistics - Theory and Methods

JF - Communications in Statistics - Theory and Methods

SN - 0361-0926

IS - 5-6

ER -