Ju Lin's AI Weblog

Evals, Explained in Plain Language

You can't just feel that an AI system got better. You need a test — and a way to grade it.

Give it a test it's never seen

what it learned fromNEWthe test

If it's already seen the answers, acing the test proves nothing. A real eval is questions it has never met before.

Score every answer

Five test questions appear one at a time, each graded with a checkmark or a cross, then a final score of 80 percent pulses in below.Q1Q2Q3Q4Q54 / 5 correct — 80%

Every answer gets checked against the right one. No vibes — just a tally at the end.

One test isn't enough

math92%coding71%safety88%chit-chat64%

Good at math doesn't mean good at everything. Run lots of different tests — one score alone can hide a lot.

Compare the old model to the new one

oldnew58% → 74%reasoning71% → 62%coding80% → 91%safety

"Better" isn't automatic. The new model can climb in reasoning and safety and still slip backwards in coding, all at once.

Watch the score over time

A line chart across five model versions draws itself from left to right, dipping at version 3 before climbing again, ending at a score of 93 percent.v1v2v3v493%

One release can dip — that's normal. What matters is the trend across many releases, on the tests that actually matter to you. Evals — that's the real name for it.