Hybrid Frame-Event Solution for Vision-Based Grasp and Pose Detection of Objects

Kyra Wang, Sihan Yang, Deepesh Kumar, Nitish Thakor

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

A key challenge in object manipulation using prosthetic hands is grasp detection and pose estimation, especially in cluttered scenes. Vision-based robotic grasping solutions typically only use conventional frame-based video cameras with high spatiotemporal redundancy, which is unsuitable for mobile platforms like prostheses with low processing power. On the other hand, while event-based dynamic vision sensors (DVS) have low spatiotemporal redundancy, their low resolution results in poor object segmentation and detection performance. In this paper we outline a novel hybrid solution inspired by the two-streams hypothesis of the neural processing of vision, utilizing both a frame-based video camera and a DVS to counter the pitfalls of both systems. By using computationally efficient object detection methods on the frame-based camera to highlight regions-of-interest (ROIs) for the DVS, we are able to perform pose estimation by computing the smallest axis of DVS events generated in the ROI. The proposed approach allows us to rapidly determine the required wrist rotation and a suitable grasp type to pick up objects using a prosthetic hand. Results on a laptop show that our method matches the accuracy of a conventional solution that employs only a frame-based video camera, while achieving 77.29% faster inference speed.

Original languageEnglish (US)
Title of host publication2020 IEEE 16th International Conference on Automation Science and Engineering, CASE 2020
PublisherIEEE Computer Society
Pages1383-1388
Number of pages6
ISBN (Electronic)9781728169040
DOIs
StatePublished - Aug 2020
Event16th IEEE International Conference on Automation Science and Engineering, CASE 2020 - Hong Kong, Hong Kong
Duration: Aug 20 2020Aug 21 2020

Publication series

NameIEEE International Conference on Automation Science and Engineering
Volume2020-August
ISSN (Print)2161-8070
ISSN (Electronic)2161-8089

Conference

Conference16th IEEE International Conference on Automation Science and Engineering, CASE 2020
CountryHong Kong
CityHong Kong
Period8/20/208/21/20

Keywords

  • Computer vision
  • Grasping
  • Neuromorphic engineering
  • Pose estimation
  • Prosthetic hand

ASJC Scopus subject areas

  • Control and Systems Engineering
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Hybrid Frame-Event Solution for Vision-Based Grasp and Pose Detection of Objects'. Together they form a unique fingerprint.

Cite this