Training Qwen to Paint Watercolors with Pure RL

Surya trained Qwen 3.5 to have taste in watercolor painting. The results are stunning.

It’s purely RL:

The system is a four-step loop, run thousands of times during training.

The model receives a prompt, something like draw a peach hibiscus in watercolour, and writes a complete p5.brush JavaScript sketch. The sketch is rendered in a sandboxed Puppeteer environment, which produces a PNG. The PNG is judged against two random reference paintings sampled from a hand-rated pool, with a separate judge model picking the better watercolour. The judgment is converted into a reward signal, GRPO updates the model, and the loop runs again.

The non-obvious choices live in what is being judged, how the judgment is made, what is in the reference pool, and how the system prompt is written.

I ran into the same problem with pixel art generation. I built a harness around it, and the results still weren’t good enough. A harness can’t save you if the model itself puts out slop. This article shows a good way to fix that for art specifically.

The reference pool isn’t big: 581 AI-generated paintings, narrowed down from 1,664 candidates. The author hand-rated them into “love-tier,” “okay,” and a supplementary batch. The candidates themselves came from Opus 4.6, GPT-5.4, and Gemini 3.1 Pro, iterating against reference photographs under a VLM judge. It’s a niche enough domain that there wasn’t enough human-made reference art to draw from, so the training data had to be manufactured this way.

A few other takeaways:

• Don’t do rubric judge, just do pairwise comparisons. The author tried a 1-10 rubric first and the scores compressed near zero. Switching to “which of these two is the better watercolour” opened up the dynamic range.
• Just handpick a golden set and compare outputs to a random one from there, instead of a reward model. The 117 love-tier paintings became the comparison pool, and every rollout got judged against those. Training an actual reward model was the next step they never got to.
• Simple prompts are better, don’t add too much detailed specs. A long reference doc in the system prompt made the model hallucinate APIs. A short, opinionated allowlist worked better than the full spec, and the generated code compressed from 13,500 tokens to under 2,000.
• To do RL on subjective work, you have to author the reward by hand, and then design it carefully enough that it generalises.