Automated video-based facial expression analysis of neuropsychiatric disorders

Peng Wang, Frederick Barrett, Elizabeth Martin, Marina Milonova, Raquel E. Gur, Ruben C. Gur, Christian Kohler, Ragini Verma

Research output: Contribution to journalArticle

Abstract

Deficits in emotional expression are prominent in several neuropsychiatric disorders, including schizophrenia. Available clinical facial expression evaluations provide subjective and qualitative measurements, which are based on static 2D images that do not capture the temporal dynamics and subtleties of expression changes. Therefore, there is a need for automated, objective and quantitative measurements of facial expressions captured using videos. This paper presents a computational framework that creates probabilistic expression profiles for video data and can potentially help to automatically quantify emotional expression differences between patients with neuropsychiatric disorders and healthy controls. Our method automatically detects and tracks facial landmarks in videos, and then extracts geometric features to characterize facial expression changes. To analyze temporal facial expression changes, we employ probabilistic classifiers that analyze facial expressions in individual frames, and then propagate the probabilities throughout the video to capture the temporal characteristics of facial expressions. The applications of our method to healthy controls and case studies of patients with schizophrenia and Asperger's syndrome demonstrate the capability of the video-based expression analysis method in capturing subtleties of facial expression. Such results can pave the way for a video-based method for quantitative analysis of facial expressions in clinical research of disorders that cause affective deficits.

Original languageEnglish (US)
Pages (from-to)224-238
Number of pages15
JournalJournal of Neuroscience Methods
Volume168
Issue number1
DOIs
StatePublished - Feb 15 2008
Externally publishedYes

Fingerprint

Facial Expression
Schizophrenia
Asperger Syndrome
Mood Disorders
Case-Control Studies
Research

Keywords

  • Affective deficits
  • Facial expression
  • Pattern classification
  • Schizophrenia
  • Video analysis

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this

Automated video-based facial expression analysis of neuropsychiatric disorders. / Wang, Peng; Barrett, Frederick; Martin, Elizabeth; Milonova, Marina; Gur, Raquel E.; Gur, Ruben C.; Kohler, Christian; Verma, Ragini.

In: Journal of Neuroscience Methods, Vol. 168, No. 1, 15.02.2008, p. 224-238.

Research output: Contribution to journalArticle

Wang, P, Barrett, F, Martin, E, Milonova, M, Gur, RE, Gur, RC, Kohler, C & Verma, R 2008, 'Automated video-based facial expression analysis of neuropsychiatric disorders', Journal of Neuroscience Methods, vol. 168, no. 1, pp. 224-238. https://doi.org/10.1016/j.jneumeth.2007.09.030
Wang, Peng ; Barrett, Frederick ; Martin, Elizabeth ; Milonova, Marina ; Gur, Raquel E. ; Gur, Ruben C. ; Kohler, Christian ; Verma, Ragini. / Automated video-based facial expression analysis of neuropsychiatric disorders. In: Journal of Neuroscience Methods. 2008 ; Vol. 168, No. 1. pp. 224-238.
@article{d9e3668211e149099bf74128b5732698,
title = "Automated video-based facial expression analysis of neuropsychiatric disorders",
abstract = "Deficits in emotional expression are prominent in several neuropsychiatric disorders, including schizophrenia. Available clinical facial expression evaluations provide subjective and qualitative measurements, which are based on static 2D images that do not capture the temporal dynamics and subtleties of expression changes. Therefore, there is a need for automated, objective and quantitative measurements of facial expressions captured using videos. This paper presents a computational framework that creates probabilistic expression profiles for video data and can potentially help to automatically quantify emotional expression differences between patients with neuropsychiatric disorders and healthy controls. Our method automatically detects and tracks facial landmarks in videos, and then extracts geometric features to characterize facial expression changes. To analyze temporal facial expression changes, we employ probabilistic classifiers that analyze facial expressions in individual frames, and then propagate the probabilities throughout the video to capture the temporal characteristics of facial expressions. The applications of our method to healthy controls and case studies of patients with schizophrenia and Asperger's syndrome demonstrate the capability of the video-based expression analysis method in capturing subtleties of facial expression. Such results can pave the way for a video-based method for quantitative analysis of facial expressions in clinical research of disorders that cause affective deficits.",
keywords = "Affective deficits, Facial expression, Pattern classification, Schizophrenia, Video analysis",
author = "Peng Wang and Frederick Barrett and Elizabeth Martin and Marina Milonova and Gur, {Raquel E.} and Gur, {Ruben C.} and Christian Kohler and Ragini Verma",
year = "2008",
month = "2",
day = "15",
doi = "10.1016/j.jneumeth.2007.09.030",
language = "English (US)",
volume = "168",
pages = "224--238",
journal = "Journal of Neuroscience Methods",
issn = "0165-0270",
publisher = "Elsevier",
number = "1",

}

TY - JOUR

T1 - Automated video-based facial expression analysis of neuropsychiatric disorders

AU - Wang, Peng

AU - Barrett, Frederick

AU - Martin, Elizabeth

AU - Milonova, Marina

AU - Gur, Raquel E.

AU - Gur, Ruben C.

AU - Kohler, Christian

AU - Verma, Ragini

PY - 2008/2/15

Y1 - 2008/2/15

N2 - Deficits in emotional expression are prominent in several neuropsychiatric disorders, including schizophrenia. Available clinical facial expression evaluations provide subjective and qualitative measurements, which are based on static 2D images that do not capture the temporal dynamics and subtleties of expression changes. Therefore, there is a need for automated, objective and quantitative measurements of facial expressions captured using videos. This paper presents a computational framework that creates probabilistic expression profiles for video data and can potentially help to automatically quantify emotional expression differences between patients with neuropsychiatric disorders and healthy controls. Our method automatically detects and tracks facial landmarks in videos, and then extracts geometric features to characterize facial expression changes. To analyze temporal facial expression changes, we employ probabilistic classifiers that analyze facial expressions in individual frames, and then propagate the probabilities throughout the video to capture the temporal characteristics of facial expressions. The applications of our method to healthy controls and case studies of patients with schizophrenia and Asperger's syndrome demonstrate the capability of the video-based expression analysis method in capturing subtleties of facial expression. Such results can pave the way for a video-based method for quantitative analysis of facial expressions in clinical research of disorders that cause affective deficits.

AB - Deficits in emotional expression are prominent in several neuropsychiatric disorders, including schizophrenia. Available clinical facial expression evaluations provide subjective and qualitative measurements, which are based on static 2D images that do not capture the temporal dynamics and subtleties of expression changes. Therefore, there is a need for automated, objective and quantitative measurements of facial expressions captured using videos. This paper presents a computational framework that creates probabilistic expression profiles for video data and can potentially help to automatically quantify emotional expression differences between patients with neuropsychiatric disorders and healthy controls. Our method automatically detects and tracks facial landmarks in videos, and then extracts geometric features to characterize facial expression changes. To analyze temporal facial expression changes, we employ probabilistic classifiers that analyze facial expressions in individual frames, and then propagate the probabilities throughout the video to capture the temporal characteristics of facial expressions. The applications of our method to healthy controls and case studies of patients with schizophrenia and Asperger's syndrome demonstrate the capability of the video-based expression analysis method in capturing subtleties of facial expression. Such results can pave the way for a video-based method for quantitative analysis of facial expressions in clinical research of disorders that cause affective deficits.

KW - Affective deficits

KW - Facial expression

KW - Pattern classification

KW - Schizophrenia

KW - Video analysis

UR - http://www.scopus.com/inward/record.url?scp=38049169004&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=38049169004&partnerID=8YFLogxK

U2 - 10.1016/j.jneumeth.2007.09.030

DO - 10.1016/j.jneumeth.2007.09.030

M3 - Article

C2 - 18045693

AN - SCOPUS:38049169004

VL - 168

SP - 224

EP - 238

JO - Journal of Neuroscience Methods

JF - Journal of Neuroscience Methods

SN - 0165-0270

IS - 1

ER -