Temporal and physical reasoning for perception-based robotic manipulation

Felix Jonathan, Chris Paxton, Gregory Hager

Research output: Contribution to journalArticlepeer-review


Accurate knowledge of object poses is crucial to successful robotic manipulation tasks, and yet most current approaches only work in laboratory settings. Noisy sensors and cluttered scenes interfere with accurate pose recognition, which is problematic especially when performing complex tasks involving object interactions. This is because most pose estimation algorithms focus only on estimating objects from a single frame, which means they lack continuity between frames. Further, they often do not consider resulting physical properties of the predicted scene such as intersecting objects or objects in unstable positions. In this work, we enhance the accuracy and stability of estimated poses for a whole scene by enforcing these physical constraints over time through the integration of a physics simulation. This allows us to accurately determine relationships between objects for a construction task. Scene parsing performance was evaluated on both simulated and realworld data. We apply our method to a real-world block stacking task, where the robot must build a tall tower of colored blocks.

Original languageEnglish (US)
JournalUnknown Journal
StatePublished - Oct 11 2017

ASJC Scopus subject areas

  • General

Fingerprint Dive into the research topics of 'Temporal and physical reasoning for perception-based robotic manipulation'. Together they form a unique fingerprint.

Cite this