Turn on sound

Once upon a time, learning from demonstration of dynamic tasks was really annoying to get working. But today, thanks to the magic of agentic programming, it can be surprisingly straightforward.

Many current robot demos show amazing capabilities but we often only see robots perform quasi-static behaviors. Quasi-static means that the robot can be paused and resumed at any time without the physics of the world taking over and causing problems. Often a key element of the robot training pipeline is human teleoperation; either done directly with the robot or through robot gripper hands such as UMI. But unfortunately, doing dynamic tasks like juggling using teleoperation is hard.

Motion capture provides an easier way to capture dynamic behaviors directly since doing hard tasks like cutting a tree would be difficult to demonstrate through a gripper interface. Today there are a number of solutions which aim to estimate human hand and body poses such as GVHMR or HaMeR. And a number of subsequent works which aim to retarget that human motion to humanoid robots to achieve dynamic behaviors (PHP Parkour). These examples are often limited to locomotion as retargeting of manipulation behaviors is not fully there yet (but there are a number of promising results: RewardAI, Unitree).

We found that if the robot is able to track just the demonstration hand motion well enough, it can actually perform some complex dynamic manipulation skills open-loop without any learning. We capture a human demonstration of a dynamic task such as cracking a whip¹, then generate a robot trajectory to track the hand motion, and execute open-loop tracking of the trajectory on the robot. Today this pipeline can be nearly fully automatically implemented with GPT-6 Astra. Below we show a two example dynamic manipulation tasks: cracking a whip and lassoing a cleat.²

Cracking a whip

Realtime 1× Slow 1/8×

We capture the human motion using Vicon motion capture with retroreflective markers on the whip handle (additional markers are placed on the whip to track the whip motion):

Realtime 1× Slow 0.25×

Drag to rotate and scroll to zoom.

A robot trajectory is then generated to track the demonstration hand motion (blue). The tracking is not exact since the robot trajectory (yellow) must obey joint velocity and dynamics limits.

Realtime 1× Slow 0.25×

Drag to rotate and scroll to zoom.

Finally the robot can track that generated trajectory to crack a whip

Realtime 1× Slow 1/8×

This process works with different types of whip crack motions such as a front cattleman:

Realtime 1× Slow motion

Sometimes even if the robot can’t track the demonstration trajectory perfectly the robot is still able to crack the whip:

Lassoing a cleat

Realtime 1× Slow 0.25×

Drag to rotate and scroll to zoom.

Realtime 1× Slow motion

Why are dynamic tasks hard to teleop?

Teleop limits the capabilities of the robot primarily in its ability to produce high velocity motions since robot motions which can achieve high velocity are difficult to generate online since a controller would need to take into account the full motion to effectively actuate the motors to not violate joint and power limits (velocity, torque, current draw, etc.).

For offline data capture where a human demonstrator is performing a task through a data capture glove such as UMI, Koala, etc. it is then possible to generate robot motions to track fast hand trajectories, but it is often difficult for a demonstrator to actually perform dynamic behaviors through the capture interface due to its weight and bulk.

How to automatically track a demonstration?

This process often used to be a tedious endeavor involving careful implementation of complex solvers, dynamics, and data management. But in the age of modern agentic coding tools, GPT-6 Astra can automatically implement retargeting pipelines from a single prompt by pulling from and combining the many wonderful open source tools created by the robotics community (drake, casadi, pinocchio, pink, mjlab, etc.).

Given a hand trajectory, the goal of retargeting is to generate a robot trajectory which tracks the motion of the hand as accurately as possible. If we didn’t care about the speed of the hand then this can be performed using the many inverse kinematics methods by either sampling many intermediate points and generating feasible joint configurations (then timing can be done with TOPPRA) or via differential inverse kinematics to smoothly track a hand path (great tutorial on IK). However, these methods do not take into account the robot dynamics. On the other hand, methods like trajectory optimization and reinforcement learning have been used to achieve dynamically feasible trajectories for high speed motions but can get stuck in local minima or be unable to track the demonstration accurately. One solution is to simply combine the methods by warm-starting the trajectory optimization with a solution generated by an inverse kinematics solver.

In addition to the robot trajectory generation, Astra was able to implement the required systems for tracking the trajectory from automatically collecting calibration data, generating a dynamics model and implementing an inverse dynamics controller.

What if the robot can’t track the demonstration?

Sometimes the high speed motion performed by the demonstrator is too fast or in a kinematic path is not feasible given the robot’s morphology and joint limits. We also explored this issue in past work: https://flying-knots.github.io/, where the robot is unable to track the demonstration trajectory accurately but with a simple dynamics model and less than 10 trials on the hardware the robot can learn to perform the task.

Why do we care about dynamic manipulation?

Because it is fun!

But I do have to call out the nice paper by Matt Mason and Kevin Lynch giving much better reasoning.

Footnotes