Measuring the effect
Did an agent improve because its examples were more useful, or because it got more labels or easier tests? A fair comparison separates these possibilities.
Count every test episode
An agent may finish before the scheduled action change. Excluding it would remove a success and leave the methods with different test cases. Intention-to-treat scoring counts every assigned episode, whether the change happens or not.
Check this with a simulation
The true simulated difference is . Counting all episodes gives bias ; counting only changed episodes gives . Bias measures average overestimation or underestimation.
Match both labels and practice
Match the label count and the amount of training. Otherwise, a gain could reflect more supervision or more practice, rather than better-chosen situations.
What happened with twice the labels?
Doubling the budget to recovery labels gave success, versus before. One run cannot establish a benefit.
Give both methods the same starting point
Training varies between runs. A paired comparison starts both methods from the same network and tests them on the same mazes. Comparing their success within each run removes some shared variation.
Why pairing can give a clearer answer
In this six-run simulation, pairing detects a five-point effect in of experiments, versus without pairing. This depends on the simulation’s assumptions.
Choose the test before seeing the answer
Choose the scenarios, scoring rules, and useful-effect threshold before seeing the results. A file fingerprint, or hash, lets us detect later edits to those rules. Try changing a value below.
Continue to the study
The study combines these choices: equal budgets and training, paired runs, and a test fixed in advance.