Training Qwen to Paint Watercolors with Pure RL

2026-08-24T00:00:00+12:00

Surya trained Qwen 3.5 to have taste in watercolor painting. The results are stunning.

It’s purely RL:

The system is a four-step loop, run thousands of times during training.

The model receives a prompt, something like draw a peach hibiscus in watercolour, and writes a complete p5.brush JavaScript sketch. The sketch is rendered in a sandboxed Puppeteer environment, which produces a PNG. The PNG is judged against two random reference paintings sampled from a hand-rated pool, with a separate judge model picking the better watercolour. The judgment is converted into a reward signal, GRPO updates the model, and the loop runs again.

The non-obvious choices live in what is being judged, how the judgment is made, what is in the reference pool, and how the system prompt is written.

I ran into the same problem with pixel art generation. I built a harness around it, and the results still weren’t good enough. A harness can’t save you if the model itself puts out slop. This article shows a good way to fix that for art specifically.

The reference pool isn’t big: 581 AI-generated paintings, narrowed down from 1,664 candidates. The author hand-rated them into “love-tier,” “okay,” and a supplementary batch. The candidates themselves came from Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, iterating against reference photographs under a VLM judge. It’s a niche enough domain that there wasn’t enough human-made reference art to draw from, so the training data had to be manufactured this way.

A few other takeaways:

Don’t do rubric judge, just do pairwise comparisons. The author tried a 1-10 rubric first and the scores compressed near zero. Switching to “which of these two is the better watercolour” opened up the dynamic range.
Just handpick a golden set and compare outputs to a random one from there, instead of a reward model. The 117 love-tier paintings became the comparison pool, and every rollout got judged against those. Training an actual reward model was the next step they never got to.
Simple prompts are better, don’t add too much detailed specs. A long reference doc in the system prompt made the model hallucinate APIs. A short, opinionated allowlist worked better than the full spec, and the generated code compressed from 13,500 tokens to under 2,000.
To do RL on subjective work, you have to author the reward by hand, and then design it carefully enough that it generalises.

Ju Lin's AI Weblog: Reinforcement-Learning

Training Qwen to Paint Watercolors with Pure RL