Chapter 04 of 08 · teaching experiments

Two ways to learn to act

Try things and keep what pays, or copy someone who already knows. This study is often mistaken for the former. It is the latter, and the distinction is worth earning with running code.

The two objectives, one line each

Reinforcement learning maximizes expected return through interaction: max_π 𝔼[ Σ_t R(s_t, a_t) ] learning from reward alone. Imitation learning fits the policy to labelled expert decisions: min_π 𝔼[ ℓ(π(o_1:t), a*_t) ] supervised learning on (observation history, expert action) pairs. The study is pure imitation: no gradient anywhere depends on reward. Reward is only read at evaluation time, to measure success.

Real reinforcement learning, where it belongs

Tabular Q-learning, implemented from scratch, on a tiny fully observable room: the state (agent position and direction, with states ever encountered) is read directly from the simulator, and the greedy policy reaches the goal in steps after a few hundred episodes. RL genuinely works when the state is visible and enumerable and reward is reachable.

None of that recipe carries over here: every seed generates a fresh world no table can enumerate (chapter 02 counted distinct observations in a few hundred episodes), the state hides behind a 7×7 window, success depends on a sentence, and reward arrives only at the end. Deep RL is not impossible: random walking already finds reward sometimes (). But the study is not asking an RL question. It asks a supervision-economics question: which labels help more. Behavior cloning is the controlled substrate for answering it.

The first cloned policies

expert demonstrations ( labelled steps), cloned into two architectures with identical data and optimizer, three seeds each: a memoryless policy, and the recurrent one the study uses. Two classic lessons fall out.