The Core Idea
Language makes robot programming accessible, but instructions can be ambiguous and LLM-generated plans can be difficult to validate. SHRIMP keeps people in the loop by making the plan visible, editable, and testable before it executes in the real-world.
Abstract
As collaborative robots have entered domains such as manufacturing, agriculture, and healthcare, programming or adapting robot behavior typically requires robotic expertise that most end users lack. Natural language lowers this barrier. Recent advancements in large language models (LLMs) have made it feasible to translate natural language into robot task plans. However, language-based task specification can suffer from semantic ambiguity, and generative models lack transparency for how language instructions are translated into robot actions, making it difficult for users to validate plans before execution. To address these issues, we introduce SHRIMP: Simulation-driven Human-in-the-loop Refinement Interface for Manipulation Planning. SHRIMP allows users to automatically generate a hierarchical robot primitive plan using natural language and iteratively revise their plan through re-prompting and explicit correction. At each revision, SHRIMP allows users to validate their plan in simulation, and once satisfied, execute it on the physical robot. Through a user study involving participants planning tabletop kitchen tasks (N = 35), we validate that SHRIMP improves perceived control and enhances robot transparency.
System Design
SHRIMP coordinates planning, simulation, perception, and robot control. SHRIMP translates a user's natural-language task description and a tracked representation of the physical scene into a hierarchical robot plan. The hierarchical plan contains sequences of high-level primitives composed of parameterized low-level robot primitives. Before simulation or execution, the plan undergoes validation checks to confirm valid grasps and reduce robot-robot and robot-object collisions. Users can then inspect and test the plan in a physics-based digital twin, step through individual primitives, and refine the plan through re-prompting, primitive editing, or parameter adjustment. Revised plans are stored in Task History, and the final plan can be executed on the physical robot.
Evaluation
Participants used SHRIMP to plan tabletop kitchen tasks. The study examined whether iterative, simulation-backed refinement helps non-experts understand and shape a robot's behavior.
Greater perceived control
Iterative revision gave participants more control over how their instructions became robot behavior.
Enhanced transparency
Exposing the hierarchical plan and its simulated outcome helped participants inspect and understand the robot's intended actions before execution.
These findings show that simulation-driven refinement can reduce ambiguity in language-based robot programming while preserving user control over the final plan.