Pretraining a Mini Kimi K3 for $252

Fri, 21 Aug 2026 00:00:00 +0000

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35. Rather than simplify the architecture, they kept Kimi K3’s MoE and attention design intact.

Some pushback from the LocalLLaMA discussion is worth noting:

5B tokens is probably too little for a 1B model. The authors agree the run was budget constrained.
Beating GPT-2 isn’t particularly meaningful when Mini K3 has roughly 10× the parameters.
MoE at this scale is debatable. A smaller dense model trained on more tokens would likely be better if the goal was capability.

For $252, they worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.

About

Mon, 01 Jan 0001 00:00:00 +0000

about

Ju Lin's AI Engineering Notebook

Pretraining a Mini Kimi K3 for $252

About