Orientation estimation and grasp type detection of household objects for upper limb prostheses with dynamic vision sensor

Although the past decade has seen important advances in prosthetic technologies, grasping household objects with an artificial hand still requires significant skill and effort for an amputee to regulating hand behaviour. A solution to this problem is to automate the process by using vision sensors that determine the object's orientation and optimal grasp procedure. In this paper, we use a neuromorphic dynamic vision sensor (DVS) to assist amputees with object grasping. Event-driven sensors such as the DVS have gained popularity in recent years as an alternative to conventional frame-based sensors due to their low-power consumption and low-latency. Here, we use event data from a DVS to find a grasp-appropriate orientation for the object and subsequently its optimal grasp type. Our estimation technique exploits general assumptions such as object symmetry and grasps preference to be along the smallest major dimension of an object. The grasp type is determined through a combination of multiple convolutional neural network (CNN) classifiers. We evaluated our grasp estimation methodology on a set of 20 household objects. The results of this study show that 96.25% of the estimated orientations were within ±10° of the actual orientations. In addition, our grasp detection method yielded a 99.47% accuracy on unseen object classes.