Visual robot task planning

Chris Paxton, Yotam Barnoy, Kapil Katyal, Raman Arora, Gregory Hager

Research output: Chapter in Book/Report/Conference proceedingConference contribution

Abstract

Prospection is key to solving challenging problems in new environments, but it has not been deeply explored as applied to task planning for perception-driven robotics. We propose visual robot task planning, where we take in an input image and must generate a sequence of high-level actions and associated observations that achieve some task. In this paper, we describe a neural network architecture and associated planning algorithm that (1) learns a representation of the world that can generate prospective futures, (2) uses this generative model to simulate the result of sequences of high-level actions in a variety of environments, and (3) evaluates these actions via a variant of Monte Carlo Tree Search to find a viable solution to a particular problem. Our approach allows us to visualize intermediate motion goals and learn to plan complex activity from visual information, and used this to generate and visualize task plans on held-out examples of a block-stacking simulation.

Original languageEnglish (US)
Title of host publication2019 International Conference on Robotics and Automation, ICRA 2019
PublisherInstitute of Electrical and Electronics Engineers Inc.
Pages8832-8838
Number of pages7
ISBN (Electronic)9781538660263
DOIs
StatePublished - May 1 2019
Event2019 International Conference on Robotics and Automation, ICRA 2019 - Montreal, Canada
Duration: May 20 2019May 24 2019

Publication series

NameProceedings - IEEE International Conference on Robotics and Automation
Volume2019-May
ISSN (Print)1050-4729

Conference

Conference2019 International Conference on Robotics and Automation, ICRA 2019
CountryCanada
CityMontreal
Period5/20/195/24/19

ASJC Scopus subject areas

  • Software
  • Control and Systems Engineering
  • Artificial Intelligence
  • Electrical and Electronic Engineering

Fingerprint Dive into the research topics of 'Visual robot task planning'. Together they form a unique fingerprint.

  • Cite this

    Paxton, C., Barnoy, Y., Katyal, K., Arora, R., & Hager, G. (2019). Visual robot task planning. In 2019 International Conference on Robotics and Automation, ICRA 2019 (pp. 8832-8838). [8793736] (Proceedings - IEEE International Conference on Robotics and Automation; Vol. 2019-May). Institute of Electrical and Electronics Engineers Inc.. https://doi.org/10.1109/ICRA.2019.8793736