Chapter 02 of 08 · teaching experiments

Seeing less than there is

A decision process is the minimal mathematics of acting over time. The one distinction that shapes this whole study is whether the agent gets to see where it actually is.

The formal object, in six symbols

A Markov decision process is a tuple (S, A, T, R): states, actions, a transition rule, a reward. A partially observable MDP adds an observation space and an observation function O : S → Ω: the agent never receives the state s, only o = O(s). Our task maps exactly onto this: the full grid and agent pose are the state; the 7×7 crop, the view direction, and the mission are the observation; reward is terminal-only; the horizon is the -step limit. Given the reset seed, the dynamics are deterministic, and all randomness sits in world generation.

Aliasing: the same input, two different right answers

Partial observability is not an abstract worry; it is countable. Across expert episodes ( visited states), the states collapse into distinct observations. of those observation classes are aliased: byte-identical observations arising from provably different worlds, and of them carry conflicting expert actions.

Two different mazes converge to the same bytes.

Two maze renderings flanking one shared observation; the expert recommends left in one world and forward in the other. — The frozen moment: identical observation, different optimal actions. This is the earliest such pair in rollout order.

The consequence

Any memoryless policy, meaning any function from single observations to actions, must disagree with the expert on at least of visited states on this distribution, no matter how it is trained. Memory is not an architectural taste; it is required for optimality. This number returns twice: when the first cloned policies are compared (04) and when recurrence is ablated (05).

The alternatives, and why we refuse them

MiniGrid can simply hand the policy the entire grid, dissolving the problem into an MDP, but no physical agent observes the world state, and this study is about acting under realistic perception. Frame stacking buys a fixed window of memory; belief states are exact but require a known world model. The study's choice, learned memory in a recurrent network, is built in chapter 05.

The world, the 7×7 partial view, and the rejected fully observable alternative side by side. — With full observability, recovery would be a lookup instead of a skill.

Summary

The task is a POMDP by construction and by measurement: real aliased states with conflicting correct actions exist, so any policy worth training will need memory, and a teacher who can see more than the student.