Keyword spotting using human electrocorticographic recordings

Griffin Milsap, Maxwell Collard, Christopher Coogan, Qinwan Rabbani, Yujing Wang, Nathan E Crone

Research output: Contribution to journalArticle

Abstract

Neural keyword spotting could form the basis of a speech brain-computer-interface for menu-navigation if it can be done with low latency and high specificity comparable to the “wake-word” functionality of modern voice-activated AI assistant technologies. This study investigated neural keyword spotting using motor representations of speech via invasively-recorded electrocorticographic signals as a proof-of-concept. Neural matched filters were created from monosyllabic consonant-vowel utterances: one keyword utterance, and 11 similar non-keyword utterances. These filters were used in an analog to the acoustic keyword spotting problem, applied for the first time to neural data. The filter templates were cross-correlated with the neural signal, capturing temporal dynamics of neural activation across cortical sites. Neural vocal activity detection (VAD) was used to identify utterance times and a discriminative classifier was used to determine if these utterances were the keyword or non-keyword speech. Model performance appeared to be highly related to electrode placement and spatial density. Vowel height (/a/ vs /i/) was poorly discriminated in recordings from sensorimotor cortex, but was highly discriminable using neural features from superior temporal gyrus during self-monitoring. The best performing neural keyword detection (5 keyword detections with two false-positives across 60 utterances) and neural VAD (100% sensitivity, ~1 false detection per 10 utterances) came from high-density (2 mm electrode diameter and 5 mm pitch) recordings from ventral sensorimotor cortex, suggesting the spatial fidelity and extent of high-density ECoG arrays may be sufficient for the purpose of speech brain-computer-interfaces.

Original languageEnglish (US)
Article number60
JournalFrontiers in Neuroscience
Volume13
Issue numberFEB
DOIs
StatePublished - Jan 1 2019

    Fingerprint

Keywords

  • Articulation
  • Automatic speech recognition (ASR)
  • Brain computer interface (BCI)
  • Electrocorticography (ECoG)
  • Keyword spotting (KWS)
  • Sensorimotor cortex (SMC)
  • Speech
  • Superior temporal gyrus (STG)

ASJC Scopus subject areas

  • Neuroscience(all)

Cite this