Tag: Moe

Flint: A Small MoE Model

Flint is a new MoE model: 39B parameters, 2.5B active per token. Architecture is 2 dense layers followed by 47 MoE layers.

Benchmarks: 54.9% on Terminal-Bench 2.1, 89.7% on SWE-bench Pro v2, 74.2% on SWE-bench Verified.

Trained from scratch (tokenizer → pretrain → midtrain → SFT → RL), giving more flexibility for fine-tuning downstream. The team built the complete training pipeline in 4 months.

Pretraining a Mini Kimi K3 for $252

Vizuara AI Labs trained a miniature Kimi K3 from scratch: 1.02B parameters, 145M active, 5B tokens, one H200, $252.35.

Not simplifying the architecture like Karpathy’s microgpt, they kept Kimi K3’s MoE and attention design intact.

That’s a surprisingly cheap way to learn pretraining (in real-world). They worked through expert collapse, data-mixing bugs, distributed-training bugs, kernels, and GPU utilization on a modern MoE architecture.

A few things worth noting:

  • 5B tokens is probably too little for a 1B model. The authors agree the run was budget constrained. So the cheap cost might due to the training stopped early.
  • Beating GPT-2 isn’t particularly meaningful when Mini K3 has roughly 10× the parameters.
  • MoE at this scale is debatable. A smaller dense model trained on more tokens would likely be better if the goal was capability.