Pretraining a Mini Kimi K3 for $252

Fri, 21 Aug 2026 00:00:00 +0000

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35. Rather than simplify the architecture (like Karpathy’s microgpt), they kept Kimi K3’s MoE and attention design intact. They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.

A few things worth noting:

5B tokens is probably too little for a 1B model. The authors agree the run was budget constrained.
Beating GPT-2 isn’t particularly meaningful when Mini K3 has roughly 10× the parameters.
MoE at this scale is debatable. A smaller dense model trained on more tokens would likely be better if the goal was capability.

That’s a surprisingly cheap way to learn pretraining.

Pretraining on Ju Lin's AI Engineering Notebook

Pretraining a Mini Kimi K3 for $252