Neuromorphic vision and tactile fusion for upper limb prosthesis control

Mark Hays, Luke Osborn, Rohan Ghosh, Mark Iskarous, Christopher Hunt, Nitish V Thakor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A major issue with upper limb prostheses is the disconnect between sensory information perceived by the user and the information perceived by the prosthesis. Advances in prosthetic technology introduced tactile information for monitoring grasping activity, but visual information, a vital component in the human sensory system, is still not fully utilized as a form of feedback to the prosthesis. For able-bodied individuals, many of the decisions for grasping or manipulating an object, such as hand orientation and aperture, are made based on visual information before contact with the object. We show that inclusion of neuromorphic visual information, combined with tactile feedback, improves the ability and efficiency of both able-bodied and amputee subjects to pick up and manipulate everyday objects. We discovered that combining both visual and tactile information in a real-time closed loop feedback strategy generally decreased the completion time of a task involving picking up and manipulating objects compared to using a single modality for feedback. While the full benefit of the combined feedback was partially obscured by experimental inaccuracies of the visual classification system, we demonstrate that this fusion of neuromorphic signals from visual and tactile sensors can provide valuable feedback to a prosthetic arm for enhancing real-time function and usability.

Original languageEnglish (US)
Title of host publication9th International IEEE EMBS Conference on Neural Engineering, NER 2019
PublisherIEEE Computer Society
Pages981-984
Number of pages4
ISBN (Electronic)9781538679210
DOIs
Publication statusPublished - May 16 2019
Event9th International IEEE EMBS Conference on Neural Engineering, NER 2019 - San Francisco, United States
Duration: Mar 20 2019Mar 23 2019

Publication series

NameInternational IEEE/EMBS Conference on Neural Engineering, NER
Volume2019-March
ISSN (Print)1948-3546
ISSN (Electronic)1948-3554

Conference

Conference9th International IEEE EMBS Conference on Neural Engineering, NER 2019
CountryUnited States
CitySan Francisco
Period3/20/193/23/19

    Fingerprint

ASJC Scopus subject areas

  • Artificial Intelligence
  • Mechanical Engineering

Cite this

Hays, M., Osborn, L., Ghosh, R., Iskarous, M., Hunt, C., & Thakor, N. V. (2019). Neuromorphic vision and tactile fusion for upper limb prosthesis control. In 9th International IEEE EMBS Conference on Neural Engineering, NER 2019 (pp. 981-984). [8716890] (International IEEE/EMBS Conference on Neural Engineering, NER; Vol. 2019-March). IEEE Computer Society. https://doi.org/10.1109/NER.2019.8716890