When a policy makes a mistake, whose states deserve the labels?
A behavior-cloned policy is trained on an expert's states but deployed on its own. Given a fixed budget of additional expert labels, this study asks one controlled question: spend them on more demonstrations, or on the states the learner actually visits after an error?
The result
One scenario, both answers
The same held-out world, the same never-trained-on corruption at the same moment. The policy trained with extra demonstrations never recovers; the policy trained on recovery labels walks to the goal.
Both arms meet the same unseen corruption at the same moment, the amber flash. This is the first unseen scenario where the two arms part ways; The Study shows the losses as well. The left pane stops two seconds after the policy stops moving, while the episode itself runs on to the step limit.
What is held equal
The comparison is only meaningful because everything except the origin of the labels is matched.
The Journey, from zero to this result
My goal here was to understand this topic deeply. Building the visualizations is what made the ideas concrete for me, and explaining each one in my own words pushed that understanding further still. That is why The Journey exists: it is the path I took, written out.
Every ingredient of the experiment is built from scratch in eight short chapters, each backed by its own small, deterministic, tested experiments. You can start with no machine-learning background and audit everything above by the end.
A maze, a mission, and a 7×7 window: what the agent can see and do.
02Seeing less than there isWhy partial observability is the problem class, with measured proof that no memoryless policy suffices.
03The teacherScripted oracles, a falsification attempt, and what an “expert label” literally is.
04Two ways to learnReal reinforcement learning on a toy task; why this study is imitation instead, and the first cloned policies.
05The networkEvery layer of the policy, and an ablation read with honest eyes.
06When copying breaksCompounding errors, corruption operators, and recovery labels: the study in miniature.
07Believing a numberIntention-to-treat, matched budgets, paired replicates, and frozen protocols, each bias demonstrated live.
08The StudyThe frozen experiment assembled from all of the above, and its result.
What this does not establish
The evidence is in silico, symbolic, discrete, and oracle-supervised: a grid world with a scripted teacher and greedy discrete control. The treatment is DAgger-style recovery-state aggregation: masked behavioral cloning, not reinforcement learning, with no physical robot, no human teachers, and no pretrained models. “Unseen” means the one other derangement of a three-action set with disjoint corruption times, not an open-world corruption family. Real robotics would additionally demand continuous control, imperfect perception, and safety constraints that this study deliberately excludes.