CAPE: Corrective Actions from Precondition Errors using Large Language Models

FMDM @ NeurIPS 2022 | LangRob @ CoRL | ICRA 2024

1Brown University, 2The University of Texas at Austin

Abstract

Extracting commonsense knowledge from a large language model (LLM) offers a path to designing intelligent robots. Existing approaches that leverage LLMs for planning are unable to recover when an action fails and often resort to retrying failed actions, without resolving the error’s underlying cause. We propose a novel approach (CAPE – Corrective Actions from Precondition Errors) that attempts to propose corrective actions to resolve precondition errors during planning. CAPE improves the quality of generated plans by leveraging few-shot reasoning from action preconditions.

Our approach enables embodied agents to execute more tasks than baseline methods while ensuring semantic correctness and minimizing re-prompting. In VirtualHome, CAPE generates executable plans while improving a human-annotated plan correctness metric from 28.89% to 49.63% over SayCan. Our improvements transfer to a Boston Dynamics Spot robot initialized with a set of skills (specified in language) and associated preconditions, where CAPE improves the correctness metric of the executed task plans by 76.49% compared to SayCan. Our approach enables the robot to follow natural language commands and robustly recover from failures, which baseline approaches largely cannot resolve or address inefficiently.

Video Summary

Methodology

CAPE uses an LLM to generate plans for tasks specified in natural language. When the agent fails to execute a step due to unsafisfied pre-conditions, we re-prompt the LLM with error information, utilizing latent commonsense reasoning and fewshot learning capabilities of LLMs to overcome pre-condition errors.

Results Summary

Quantiative Results

CAPE reduces executability error rate by ~50% and improves semantic correctness by ~20% compared to the best baselines

Qualitative Results

Qualitative results of CAPE for robot execution and in the VirtualHome environment. Re-prompting with precondition error information help resolve action failures and progress plans


BibTeX

@inproceedings{raman2024cape,
      title={{CAPE: Corrective Actions from Precondition Errors using Large Language Models}},
      author={Sundara Raman, Shreyas and Cohen, Vanya and Paulius, David and Idrees, Ifrah and Rosen, Eric and Mooney, Ray and Tellex, Stefanie},
      booktitle={2024 IEEE International Conference on Robotics and Automation (ICRA)},
      year={2024},
      note={(In Review)}
    }