Skip to experiment
SYN·CO·RE LAB AI Experiment · PSY3230

Q-Learning Gridworld

An agent that knows nothing — not even that the goal exists — learns to navigate purely from reward and punishment. Reinforcement learning is operant conditioning, implemented.
What is happening here?

The agent sees only its current cell and four actions. After each move it receives a number: −0.1 per step (effort), −10 for falling off the cliff, +10 at the goal. From these numbers alone it builds Q-values — estimates of how much future reward each action is worth in each cell. The update is driven by the prediction error: the difference between the reward it expected and what it got. Cells glow green as expected value propagates backwards from the goal, step by step, episode by episode.

For a psychologist this is familiar territory: it is Thorndike's law of effect made algorithmic, and the prediction-error term is the same quantity that midbrain dopamine neurons appear to signal (Schultz et al., 1997). The ε slider is the exploration–exploitation dilemma: with ε = 0 the agent exploits what it knows and may never find a better route; with high ε it keeps gambling — watch it hug the cliff less when ε is high, because one random step near the edge is fatal.

Experiments to try: train the agent, then move the goal (edit mode) and watch it persist with the old habit before relearning — extinction in action. Wall off its favorite route. Set ε to 0 from the start and see it satisfice on the first path it stumbles into.

edit (click/drag on grid):

Parameters

Learning curve (return per episode)