Skip to experiment
SYN·CO·RE LAB AI Experiment · PSY3230

Next-Token Predictor

A tiny language model, trained in your browser in milliseconds. Large language models do — at their core — the same thing, scaled up by a factor of billions.
What is happening here?

This page counts, for every short sequence of tokens in the training text, which token tends to come next (an n-gram model). As you type, it looks at your last few tokens and shows the probability distribution over the next one. "Generating text" means sampling from that distribution, over and over. Click a prediction to accept it — that is autocomplete; do it repeatedly and you are the sampling loop.

An LLM differs in mechanism — a neural network computing the distribution from the entire context, over word fragments rather than characters — but not in objective: it is trained to do exactly this, predict the next token. Everything that looks like reasoning, style, or personality in generated text is downstream of this one task. Watching it here, in its minimal form, is a useful inoculation against reading too much mind into fluent output — and an honest puzzle about how much capability "mere prediction" can buy at scale.

Experiments to try: shrink the context length to 1 and watch the output dissolve into gibberish; switch to word-level tokens and see more "sensible" but more parroted text; paste your own writing and watch it imitate you.

1 — Training text (edit or paste your own, then retrain)

2 — Type, and watch the model predict — or click a prediction to accept it

3 — Let it generate

Low temperature: always pick the likeliest token (repetitive, "safe"). High: sample more randomly (creative, then incoherent). The same dial exists in LLMs.