Learning from examples

An agent can learn from rewards or from examples. Here, it learns to predict the actions an expert would take.

Rewards or examples

Reinforcement learning learns from rewards. Imitation learning learns from an expert. We use behavioral cloning: predict an action, compare it with the expert’s label, and adjust the network to reduce the error.

Learning from rewards

Q-learning updates a score for each action using the rewards it receives. In this small room, it encounters states and learns a route of steps.

Evaluation return and route length over Q-learning training. — As training progresses, the agent earns more reward and takes fewer steps. This is a separate toy task.

Our study instead asks how best to use expert labels. Both methods therefore learn by behavioral cloning. Reward is used only to measure task success.

Learning from the same examples

Both networks receive the same demonstrations and action labels. One uses the current view; the other also keeps memory. Each is trained three times.

Both agents face the same new maze. This is one example; the chart summarizes many episodes.

Why copying actions is not enough

Correctly predicting actions on an expert’s route does not ensure the agent can finish a task alone. One mistake changes what it sees next. Further mistakes can then take it farther from the training examples.

Compare the two measurements

The agent without memory matches of expert actions and solves of new episodes. With memory, these figures are and . Action accuracy is measured on recorded expert steps; task success is measured while the agent chooses its own route.

Behavioral-cloning loss across optimizer updates for memoryless and recurrent policies. — Lower loss means better predictions of training labels. Task success must be tested separately.

What to look at next

Three runs give limited precision. Next, we look inside the network, then examine what happens after a wrong action.