The policy network
The network combines four inputs with memory to choose an action. Follow the data through each step below.
From inputs to an action
Follow the arrows to the action choice. Select a block for an explanation, then move the slider to inspect its output at different steps.
What the memory stores
The mission GRU reads the instruction word by word and keeps a summary for the whole episode. The policy GRU updates its memory after each observation. GRU stands for gated recurrent unit.
The previous-action input records what actually happened, including a forced change. The agent gets no warning flag or expert advice. It must decide what to do from its inputs and memory.
Does each part help?
Remove or replace one component, then train again. Here we try no memory, no instruction, no previous action, and an instruction encoder that ignores word order. All versions use the same demonstrations and training settings.
How to read this comparison
Each version was trained three times, and their results overlap. Removing the instruction does not reduce success here: the agent can visit several objects before finding the target. Reaching the goal therefore need not mean understanding the words.
Why use this architecture?
This network processes one observation at a time with a fixed-size memory. It suits this small experiment, though these comparisons do not establish a best design.
From predictions to mistakes
An action may take the agent somewhere unfamiliar. Next, we ask which examples could help it recover.