Moe

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35. Rather than simplify the architecture (like Karpathy’s microgpt), they kept Kimi K3’s MoE and attention design intact. They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture. A few things worth noting: 5B tokens is probably too little for a 1B model. The authors agree the run was budget constrained. Beating GPT-2 isn’t particularly meaningful when Mini K3 has roughly 10× the parameters. MoE at this scale is debatable. A smaller dense model trained on more tokens would likely be better if the goal was capability. That’s a surprisingly cheap way to learn pretraining.