Sensor substitution for video-based action recognition

Christian Rupprecht, Colin Lea, Federico Tombari, Nassir Navab, Gregory Hager

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

There are many applications where domainspecific sensing, such as accelerometers, kinematics, or force sensing, provide unique and important information for control or for analysis of motion. However, it is not always the case that these sensors can be deployed or accessed beyond laboratory environments. For example, it is possible to instrument humans or robots to measure motion in the laboratory in ways that it is not possible to replicate in the wild. An alternative, which we explore in this paper, is to address situations where accurate sensing is available while training an algorithm, but for which only video is available for deployment. We present two examples of this sensory substitution methodology. The first variation trains a convolutional neural network to regress real-valued signals, including robot end-effector pose, from video. The second example regresses binary signals derived from accelerometer data which signifies when specific objects are in motion. We evaluate these on the JIGSAWS dataset for robotic surgery training assessment and the 50 Salads dataset for modeling complex structured cooking tasks. We evaluate the trained models for video-based action recognition and show that the trained models provide information that is comparable to the sensory signals they replace.

Original languageEnglish (US)
Title of host publicationIROS 2016 - 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages5230-5237
Number of pages8
Volume2016-November
ISBN (Electronic)9781509037629
DOIs
StatePublished - Nov 28 2016
Event2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016 - Daejeon, Korea, Republic of
Duration: Oct 9 2016Oct 14 2016

Other

Other2016 IEEE/RSJ International Conference on Intelligent Robots and Systems, IROS 2016
CountryKorea, Republic of
CityDaejeon
Period10/9/1610/14/16

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Software
  • Computer Vision and Pattern Recognition
  • Computer Science Applications

Fingerprint Dive into the research topics of 'Sensor substitution for video-based action recognition'. Together they form a unique fingerprint.

  • Cite this

    Rupprecht, C., Lea, C., Tombari, F., Navab, N., & Hager, G. (2016). Sensor substitution for video-based action recognition. In IROS 2016 - 2016 IEEE/RSJ International Conference on Intelligent Robots and Systems (Vol. 2016-November, pp. 5230-5237). [7759769] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/IROS.2016.7759769