Partial observability
Two different situations can look identical to the agent. Earlier observations may tell it which situation it is in.
The world and the agent’s view
The state describes the whole maze and the agent’s position. The observation contains only its local view, direction, and instruction. Each action changes the state and produces a new observation.
This is called partial observability. A wall or an earlier turn may matter even when it is outside the current view.
The same observation, different expert actions
The same observation can call for different expert actions. An agent using only that observation cannot tell these cases apart. This is perceptual aliasing.
How often did this occur?
Across expert episodes, the lab recorded states and distinct observations. Of those observation groups, had conflicting expert actions.
A limit of using only the current view
Without memory, an agent must disagree with the expert on at least of these recorded states. It may still reach the goal by another route, so this is a limit on matching the expert, not on task success.
How memory helps
A recurrent network carries memory from one step to the next. Training can teach it to retain useful information from earlier observations. Having memory makes this possible, but does not guarantee it learns what to keep.
Another option is to show recent observations together, or reveal the full maze. Here we keep the partial view and use learned memory.
Where the teaching signal comes from
Memory helps distinguish situations. Expert labels teach which action to take in them. That is the next lesson.