Learning convolutional action primitives for fine-grained action recognition

Colin Lea, René Vidal, Gregory Hager

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Fine-grained action recognition is important for many applications of human-robot interaction, automated skill assessment, and surveillance. The goal is to segment and classify all actions occurring in a time series sequence. While recent recognition methods have shown strong performance in robotics applications, they often require hand-crafted features, use large amounts of domain knowledge, or employ overly simplistic representations of how objects change throughout an action. In this paper we present the Latent Convolutional Skip Chain Conditional Random Field (LC-SC-CRF). This time series model learns a set of interpretable and composable action primitives from sensor data. We apply our model to cooking tasks using accelerometer data from the University of Dundee 50 Salads dataset and to robotic surgery training tasks using robot kinematic data from the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our performance on 50 Salads and JIGSAWS are 18.0% and 5.3% higher than the state of the art, respectively. This model performs well without requiring hand-crafted features or intricate domain knowledge. The code and features have been made public.

Original languageEnglish (US)
Title of host publication2016 IEEE International Conference on Robotics and Automation, ICRA 2016
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages1642-1649
Number of pages8
Volume2016-June
ISBN (Electronic)9781467380263
DOIs
StatePublished - Jun 8 2016
Event2016 IEEE International Conference on Robotics and Automation, ICRA 2016 - Stockholm, Sweden
Duration: May 16 2016May 21 2016

Other

Other2016 IEEE International Conference on Robotics and Automation, ICRA 2016
CountrySweden
CityStockholm
Period5/16/165/21/16

Fingerprint

Time series
Human robot interaction
Cooking
End effectors
Accelerometers
Kinematics
Robotics
Robots
Sensors
Robotic surgery

ASJC Scopus subject areas

  • Software
  • Artificial Intelligence
  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Cite this

Lea, C., Vidal, R., & Hager, G. (2016). Learning convolutional action primitives for fine-grained action recognition. In 2016 IEEE International Conference on Robotics and Automation, ICRA 2016 (Vol. 2016-June, pp. 1642-1649). [7487305] Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2016.7487305

Learning convolutional action primitives for fine-grained action recognition. / Lea, Colin; Vidal, René; Hager, Gregory.

2016 IEEE International Conference on Robotics and Automation, ICRA 2016. Vol. 2016-June Institute of Electrical and Electronics Engineers Inc., 2016. p. 1642-1649 7487305.

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Lea, C, Vidal, R & Hager, G 2016, Learning convolutional action primitives for fine-grained action recognition. in 2016 IEEE International Conference on Robotics and Automation, ICRA 2016. vol. 2016-June, 7487305, Institute of Electrical and Electronics Engineers Inc., pp. 1642-1649, 2016 IEEE International Conference on Robotics and Automation, ICRA 2016, Stockholm, Sweden, 5/16/16. https://doi.org/10.1109/ICRA.2016.7487305
Lea C, Vidal R, Hager G. Learning convolutional action primitives for fine-grained action recognition. In 2016 IEEE International Conference on Robotics and Automation, ICRA 2016. Vol. 2016-June. Institute of Electrical and Electronics Engineers Inc. 2016. p. 1642-1649. 7487305 https://doi.org/10.1109/ICRA.2016.7487305
Lea, Colin ; Vidal, René ; Hager, Gregory. / Learning convolutional action primitives for fine-grained action recognition. 2016 IEEE International Conference on Robotics and Automation, ICRA 2016. Vol. 2016-June Institute of Electrical and Electronics Engineers Inc., 2016. pp. 1642-1649
@inproceedings{43e7d5aa46504facb53a797330150d8b,
title = "Learning convolutional action primitives for fine-grained action recognition",
abstract = "Fine-grained action recognition is important for many applications of human-robot interaction, automated skill assessment, and surveillance. The goal is to segment and classify all actions occurring in a time series sequence. While recent recognition methods have shown strong performance in robotics applications, they often require hand-crafted features, use large amounts of domain knowledge, or employ overly simplistic representations of how objects change throughout an action. In this paper we present the Latent Convolutional Skip Chain Conditional Random Field (LC-SC-CRF). This time series model learns a set of interpretable and composable action primitives from sensor data. We apply our model to cooking tasks using accelerometer data from the University of Dundee 50 Salads dataset and to robotic surgery training tasks using robot kinematic data from the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our performance on 50 Salads and JIGSAWS are 18.0{\%} and 5.3{\%} higher than the state of the art, respectively. This model performs well without requiring hand-crafted features or intricate domain knowledge. The code and features have been made public.",
author = "Colin Lea and Ren{\'e} Vidal and Gregory Hager",
year = "2016",
month = "6",
day = "8",
doi = "10.1109/ICRA.2016.7487305",
language = "English (US)",
volume = "2016-June",
pages = "1642--1649",
booktitle = "2016 IEEE International Conference on Robotics and Automation, ICRA 2016",
publisher = "Institute of Electrical and Electronics Engineers Inc.",

}

TY - GEN

T1 - Learning convolutional action primitives for fine-grained action recognition

AU - Lea, Colin

AU - Vidal, René

AU - Hager, Gregory

PY - 2016/6/8

Y1 - 2016/6/8

N2 - Fine-grained action recognition is important for many applications of human-robot interaction, automated skill assessment, and surveillance. The goal is to segment and classify all actions occurring in a time series sequence. While recent recognition methods have shown strong performance in robotics applications, they often require hand-crafted features, use large amounts of domain knowledge, or employ overly simplistic representations of how objects change throughout an action. In this paper we present the Latent Convolutional Skip Chain Conditional Random Field (LC-SC-CRF). This time series model learns a set of interpretable and composable action primitives from sensor data. We apply our model to cooking tasks using accelerometer data from the University of Dundee 50 Salads dataset and to robotic surgery training tasks using robot kinematic data from the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our performance on 50 Salads and JIGSAWS are 18.0% and 5.3% higher than the state of the art, respectively. This model performs well without requiring hand-crafted features or intricate domain knowledge. The code and features have been made public.

AB - Fine-grained action recognition is important for many applications of human-robot interaction, automated skill assessment, and surveillance. The goal is to segment and classify all actions occurring in a time series sequence. While recent recognition methods have shown strong performance in robotics applications, they often require hand-crafted features, use large amounts of domain knowledge, or employ overly simplistic representations of how objects change throughout an action. In this paper we present the Latent Convolutional Skip Chain Conditional Random Field (LC-SC-CRF). This time series model learns a set of interpretable and composable action primitives from sensor data. We apply our model to cooking tasks using accelerometer data from the University of Dundee 50 Salads dataset and to robotic surgery training tasks using robot kinematic data from the JHU-ISI Gesture and Skill Assessment Working Set (JIGSAWS). Our performance on 50 Salads and JIGSAWS are 18.0% and 5.3% higher than the state of the art, respectively. This model performs well without requiring hand-crafted features or intricate domain knowledge. The code and features have been made public.

UR - http://www.scopus.com/inward/record.url?scp=84977479301&partnerID=8YFLogxK

UR - http://www.scopus.com/inward/citedby.url?scp=84977479301&partnerID=8YFLogxK

U2 - 10.1109/ICRA.2016.7487305

DO - 10.1109/ICRA.2016.7487305

M3 - Conference contribution

AN - SCOPUS:84977479301

VL - 2016-June

SP - 1642

EP - 1649

BT - 2016 IEEE International Conference on Robotics and Automation, ICRA 2016

PB - Institute of Electrical and Electronics Engineers Inc.

ER -