Pretraining · Ju Lin's AI Weblog

Pretraining is the first stage of training a language model. Its main task is to predict the next token. Take “The capital of France is ___.” A pretrained model reads “The capital of France is” and predicts the next token, “Paris.”

The model makes a prediction, compares it with the real token, calculates the error, and updates its weights.

To pretrain a model, you repeat this step over and over, on a huge pile of text, code, and other data:

Animated diagram of the pretraining loop: tokens feed a prediction, the prediction is compared to the real token to get a loss, the loss updates the model’s weights, and the loop repeats.

After pretraining, the model has picked up language, facts, code, and common reasoning patterns from that data.

Pretraining is only the first stage. Compare it with the two stages that usually follow:

Stage	Goal	Data	Feedback signal
Pretraining	Learn general language patterns	Huge, mixed text and code	Next-token prediction error
Fine-tuning	Learn a specific task or format	Small, curated examples	Difference from a labeled output
RL	Learn a preferred behavior	The model’s own outputs	A reward score

Pretraining teaches a model what patterns exist in its data. Feed it a lot of code, and it gets better at code. Feed it many tool-call examples, and tool use can become a natural part of its output.