Tag: Pretraining

Pretraining

Pretraining is the first stage of training a language model. Its main task is to predict the next token. Take “The capital of France is ___.” A pretrained model reads “The capital of France is” and predicts the next token, “Paris.”

The model makes a prediction, compares it with the real token, calculates the error, and updates its weights.

To pretrain a model, you repeat this step over and over, on a huge pile of text, code, and other data:

Animated diagram of the pretraining loop: tokens feed a prediction, the prediction is compared to the real token to get a loss, the loss updates the model’s weights, and the loop repeats.

After pretraining, the model has picked up language, facts, code, and common reasoning patterns from that data.

Pretraining is only the first stage. Compare it with the two stages that usually follow:

Stage	Goal	Data	Feedback signal
Pretraining	Learn general language patterns	Huge, mixed text and code	Next-token prediction error
Fine-tuning	Learn a specific task or format	Small, curated examples	Difference from a labeled output
RL	Learn a preferred behavior	The model’s own outputs	A reward score

Pretraining teaches a model what patterns exist in its data. Feed it a lot of code, and it gets better at code. Feed it many tool-call examples, and tool use can become a natural part of its output.

1 month ago / explainer pretraining llm

Marin 535B-A23B Starts Training, in the Open

🚢 Marin 535B-A23B started training this week! As usual, the whole process is open.
Voyage plan: pretraining (80%) + midtraining (20%) on 18.75T tokens on 11 x GB200 NVL72 for ~3 months (2.7e24 FLOPs). Post-training will follow.
Before kicking off the run, we trained a 4-rung scaling ladder from 1.6B-A61M (48B tokens) to 27.7B-A1.2B (926B tokens) to debug issues, and to make a forecast of our hero run. This is by far our biggest run, so definitely expecting the unexpected.

Percy Liang, announcing the run on X.

The Marin project is a great example of being open to AI. The whole process is public: the scaling ladder they ran to debug the pipeline before committing GPU-months to the real thing, the exact token counts, the exact FLOPs, even the admission that they’re “expecting the unexpected” on their biggest run yet.

Most labs treat a run like this as a trade secret until the model ships. It’s great we see another public model build. The last public run at this scale was BLOOM, BigScience’s 176B model. Marin’s 535B total parameters (23B active) puts it well past that, the biggest public pretraining run that I’m aware of.

1 month ago / pretraining field-notes training

Pretraining a Mini Kimi K3 for $252

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35.

Not simplifying the architecture like Karpathy’s microgpt, they kept Kimi K3’s MoE and attention design intact.

That’s a surprisingly cheap way to learn pretraining (in real-world). They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.

A few things worth noting:

5B tokens is probably too little for a 1B model. The authors agree the run was budget constrained. So the cheap cost might due to the training stopped early.
Beating GPT-2 isn’t particularly meaningful when Mini K3 has roughly 10× the parameters.
MoE at this scale is debatable. A smaller dense model trained on more tokens would likely be better if the goal was capability.

1 month ago / pretraining moe kimi field-notes