Chapter 06 of 08 · teaching experiments

When copying breaks

Behavior cloning trains on the expert's states and deploys on its own. The first wrong action leads somewhere the demonstrations never covered, and whatever happens there is unsupervised. This chapter makes that spiral measurable, then previews the study's remedy comparison in miniature.

The failure mode, sketched and then measured

Ross and Bagnell's classical analysis says a per-step error rate ε can cost on the order of ε·T² over a horizon T, precisely because mistakes change the state distribution. Here is that argument as data: one corrupted executed action, and the drift it causes.

Left: success drops when one action is corrupted, more for later corruption times. Right: corrupted rollouts drift about twice as far from the expert path as clean ones. — Left: one corrupted action costs real success (delivered episodes, by corruption time). Right: distance to the expert's route after the corruption; clean rollouts drift slowly (alternative valid paths), corrupted ones roughly twice as far.

Corruption operators, and why the action set is frozen

A corruption must change every action it touches, otherwise some corruptions would be no-ops. An operator is therefore a derangement (a fixed-point-free shuffle) of the action set, and for three actions exactly two exist, the two cycles, each the other's inverse. The study uses one during recovery collection and holds the other out as the unseen operator. Change the action set and this whole operator family changes with it; enlarge it and the oracle starts recommending actions outside the set (with closed doors the bot emits toggle, the pilot discovery that fixed doors_open: true in the contract). That is why the action set is a frozen contract field, not a tuning knob.

Two remedies, one budget

Both remedies start from the same trained base policy and receive exactly additional oracle labels. The extra arm spends them on fresh demonstrations. The recovery arm spends them where the learner actually goes: roll the base policy out, corrupt one action, and let the still-synchronized oracle label the next few states the policy visits.

One fairness leak is deliberately left open here and measured: an extra-demo batch carries new labels per update against for sparse recovery windows. The study closes exactly this leak with one-target-per-window training items, chapter 07 explains why that matters.

The mini-study

The base policy and the recovery-trained policy, identical world, identical unseen-operator corruption.

Step through the recovery policy's episode yourself; the amber frame marks the corrupted execution; watch the previous-action channel's purpose become visible:

A scenario where the recovery-trained policy fails anyway. The losses belong here as much as the wins.

How to read the mini-study

Three replicates, and the per-replicate recovery-minus-extra differences on the unseen slice are , and the sign is not even fixed at this scale. Two reasons, both instructive: a single corruption dents this strong base policy by only → on the matched slice (little headroom for any remedy), and small unfrozen runs simply cannot answer this question. Turning those two lessons into design requirements is the next chapter; the confirmatory answer is the study.

Summary

Why cloned policies fail under their own mistakes, what a corruption operator is and why the action set is frozen, what recovery labels are, and why a three-replicate preview settles nothing. Everything now hinges on measuring honestly.