Ju Lin's AI Weblog

Vibe Code in a Secured Way

2026-10-05T00:00:00+13:00

AI coding is powerful. It is also risky if you are not cautious.

The AI does the work. You are responsible for everything.

The bottom-line skill for every engineer nowadays is the judgment to recognize when the AI is performing risky operations.

Know What You’re Exposing

Coding sessions can be stored remotely. LLM requests can be routed through proxies. It’s a good practice to limit what the agent and the LLM model can see.

This is not just about credentials.

When you work on any data or codebase, ask yourself: is this data appropriate for the AI to see?

When it involves customer data or personal information, think carefully about whether redaction is needed. The AI should not have visibility into data it does not need to accomplish the task.

The AI Can Ignore Your Instructions

You can ask the AI not to look at something. It can ignore you anyway.

More dangerous: the AI can be spoofed through prompt injection and tricked into performing malicious instructions.

This risk magnifies when you open up the entire internet to the agent. The internet contains everything, and not all of it is trustworthy.

Trust Your Tools and Dependencies

Only use the tools or MCP servers that you trust.

Secure your dependencies. Use tools like Dependabot to scan for vulnerabilities. Monitor supply chain attacks—remember the LiteLLM PyPI attack? That kind of compromise can propagate through your entire toolchain.

If you install an untrusted package, you are giving the AI access to whatever that package can do. And the AI might use it.

The Practical Defense

When an AI agent is running:

Audit what data it has access to. Is it necessary?
Audit what tools it can call. Are they all trustworthy?
Audit what network access it has. Does it need the internet?
Audit what code it writes. Does it make risky calls?
Audit what credentials it carries. Are they overly permissive?

You cannot trust the AI to limit itself. You have to enforce limits at the boundary.

The most dangerous sessions are the ones where you trust the AI completely.

Rio 0.7.1 released

2026-10-05T00:00:00+13:00

Rio 0.7.1 is out. This is a big change to Rio’s core.

Thanks to Rulin Shao’s work on context management. There is a new way for managing context. Rio took it one step further: instead of treating context as a file to edit, treat it as a mutable, executable document. Jupyter notebook is picked because it already has a stable specification and a runtime.

@RulinShao Nice work. I fit CLM in for coding agent Rio: 1. LLM context is .ipynb. 2. LLM changes a cell in each turn. - codemode natively supported. 3. Instead of CLM's file updates, Rio uses json patch inspired by SKILL.state.
Fewer turns, less tokens https://t.co/YYJDYiSmpl https://t.co/SyxBwPF95N
— soasme (@soasme) October 3, 2026

Agent Codemode Explained

2026-10-01T00:00:00+13:00

Code Mode: Moving Agent Control Flow Out of the LLM

Most agents still use tools one call at a time.

The model decides to call a tool. The runtime executes it. The result is appended to the conversation. The model runs again and decides what to do next.

For a short task this works well.

For a task that needs twenty API calls, filtering, retries, a loop, or several dependent operations, the design becomes expensive:

LLM
 ↓
tool A
 ↓
LLM
 ↓
tool B
 ↓
LLM
 ↓
tool C
 ↓
LLM

Each edge through the LLM costs inference time. Tool results also accumulate in the context even when most of the data is only needed to decide the next call.

Code Mode changes this boundary.

Instead of asking the model to emit every tool call, give it one tool that executes a small program:

LLM
 ↓
program
 ├─ tool A
 ├─ tool B
 ├─ tool C
 └─ local filtering
 ↓
LLM

The program becomes the control plane for a section of the agent run.

This idea now appears in Cloudflare Code Mode, Anthropic’s programmatic tool calling, PydanticAI Code Mode, Hugging Face smolagents, and, as of September 29, 2026, Pi.

They share the same basic observation:

Tool calling is an unusually bad programming language.

JSON tool calls can express:

call X with Y

A normal programming language can express:

call X for every Y
call X and Z concurrently
retry X if it fails
filter the result
call Z only if condition Q is true
store this value and reuse it later
return only these five fields

The difference is larger than syntax.

Code Mode changes where computation happens.

The normal tool loop

Suppose an agent needs to find all unhealthy services and inspect their recent deployments.

With normal tool calling it might do:

LLM → list_services()
LLM ← 300 services

LLM → get_health(service_1)
LLM ← result

LLM → get_health(service_2)
LLM ← result

...

A capable model can issue several calls in parallel, but the model still has to construct the calls and receive their results.

A better tool might support bulk operations. But now every tool designer has to predict every future composition:

get_health_for_services(...)
get_unhealthy_services(...)
get_unhealthy_services_with_deployments(...)

This is where Code Mode becomes interesting.

The model can instead write something equivalent to:

services = await list_services()

health = await gather(
 *[get_health({"id": s["id"]}) for s in services]
)

bad = [
 s for s, h in zip(services, health)
 if h["status"] != "healthy"
]

deployments = await gather(
 *[get_deployments({"service_id": s["id"]}) for s in bad]
)

return [
 {
 "service": s["name"],
 "deployment": d[0]
 }
 for s, d in zip(bad, deployments)
]

The LLM sees the task once and writes the control flow once.

The intermediate 300 service records and hundreds of health responses do not need to become conversation history.

That is the important part.

Code Mode is not mainly a prettier way to call tools.

It is a way to move intermediate computation out of model context.

Cloudflare: code as a compact API plan

Cloudflare has probably made the clearest argument for Code Mode.

Its problem is unusually visible. The Cloudflare API has more than 2,500 endpoints. Exposing every API endpoint as an MCP tool means shipping a huge collection of JSON schemas into the model context before the agent does anything useful.

Cloudflare’s Code Mode MCP server instead exposes essentially two operations: search for an API surface, then execute code against the discovered typed API.

Cloudflare reports that its whole API can be exposed in roughly 1,000 tokens this way. Its comparison estimates 1.17 million input tokens if the equivalent API were represented as ordinary MCP tool definitions.

This solves the first Code Mode problem:

too many tools

There is a second problem:

too much intermediate data

Cloudflare describes generated code as a compact plan. Calls, filtering and transformations happen inside the execution environment, and only selected output returns to the model.

Their current interface exposes primitives such as:

codemode.search(...)
codemode.describe(...)
codemode.step(...)
codemode.run(...)

search() performs progressive discovery. describe() loads the detailed interface only when needed. step() records work that needs replay semantics. run() executes reusable snippets.

This is an important refinement.

A naive Code Mode implementation still puts every available function declaration in the system prompt:

one code tool
+
10,000 function declarations

That fixes intermediate results but not tool-definition cost.

Cloudflare instead makes the API itself lazy.

The model starts with a small discovery interface and pulls schemas when it needs them.

Conceptually:

 ┌──────── search
LLM → code│
 ├──────── describe
 │
 ├──────── API A
 ├──────── API B
 └──────── API C

The tool catalog becomes something closer to a filesystem or symbol table than a prompt.

That is likely the right abstraction for very large tool spaces.

Cloudflare’s sandbox matters

Once the model can write arbitrary JavaScript, eval() is not an acceptable runtime.

Cloudflare executes generated code in isolated Workers. Its MCP design blocks direct outbound access by default; generated code reaches the outside world through capabilities supplied by the host. Credentials remain outside the generated program.

The distinction is useful:

code execution capability
 ≠
external authority

The sandbox can be computationally expressive while having almost no ambient authority.

The program may know how to write:

await dns.deleteZone(...)

but the actual authority still belongs to the host connector.

This means authorization does not have to be implemented by trusting generated code.

It remains at the tool boundary.

Cloudflare’s durable runtime goes further. It records execution history and can pause generated code for approval, then replay completed work and continue the program.

That turns Code Mode from “run some generated JavaScript” into an agent runtime primitive.

Anthropic: programmatic tool calling

Anthropic independently arrived at almost exactly the same model.

Their current term is programmatic tool calling.

Claude writes Python inside its code execution environment. Tool calls from the program cross back to the host. When a tool returns, execution resumes. Intermediate results remain inside the execution environment instead of being inserted into Claude’s context.

The execution model looks like this:

Claude generates Python

 │
 ▼

sandbox
 │
 ├── tool()
 │ │
 │ └──── host executes tool
 │
 ├── process result locally
 │
 ├── tool()
 │
 └── print small result

 │
 ▼

Claude continues

This is not equivalent to asking Claude to generate a Python script and running the script after the model finishes.

The interesting part is the suspended host call.

The Python program is effectively an orchestration language around external capabilities.

Anthropic reports useful measurements here. On its BrowseComp and DeepSearchQA experiments, programmatic tool calling improved performance by an average of 11% while using 24% fewer input tokens. On an internal 75-tool project-management benchmark it reports roughly 38% lower billed input tokens with unchanged task accuracy. But on τ²-bench tasks dominated by one or two sequential calls, it cost about 8% more.

That last number is important.

Code Mode is not free.

For:

call tool
return answer

generating a program and starting a code environment is extra work.

Code Mode becomes useful when the code replaces enough model-mediated control flow to pay for itself.

Pi: Code Mode moves into the harness

Pi is a more interesting implementation for coding agents because Code Mode is not only an API optimization.

It is integrated into the agent harness.

A large Pi change was merged on September 29, 2026. The implementation runs model-generated JavaScript in a QuickJS VM compiled to WebAssembly. The script can call Pi tools as async functions. Nested calls do not enter the LLM context individually; only explicit script output and the final return value do.

The standalone package describes the capability directly:

const source = await tools.read({ path: "package.json" })
text(JSON.parse(source).name)

Every execution gets a QuickJS VM in a worker. The VM has its own WASM memory and no general route back into the Node host except the explicitly installed host-call bridge.

This makes Pi’s implementation worth studying.

Pi already had very powerful tools:

read
write
edit
bash

For coding work, bash alone is close to universal.

So why add Code Mode?

Because universality is not the same as a good agent interface.

Consider:

cat package.json | jq ...

versus:

const p = await tools.read({path: "package.json"})
const pkg = JSON.parse(p)

Both work.

But the second form can combine normal agent tools, MCP tools and future host capabilities under one runtime, without turning every operation into shell text.

More importantly, nested calls remain visible to the harness.

Pi routes nested calls through the normal validation and tool hooks. They carry a parentToolCallId, and nested calls are recorded on the parent tool result.

This gives a useful structure:

codemode call
│
├── read
├── grep
├── MCP: search
└── bash

instead of hiding the entire operation behind an opaque process.

That matters for observability, permissions and evaluation.

Pi’s tool exposure model

Pi also exposes a design problem that smaller Code Mode demos can avoid:

Which tools should the model see directly?

Pi now distinguishes different forms of tool exposure and supports two useful Code Mode configurations.

In codemode.mode = "on", normal tools may remain directly visible while Code Mode can also call them.

In codemode.mode = "only", the model sees Code Mode as the primary interface and calls ordinary tools through generated code instead.

This gives two different agent architectures.

Hybrid

LLM
 ├── read
 ├── edit
 ├── bash
 └── codemode
 ├── read
 ├── edit
 └── MCP tools

Code-only

LLM
 └── codemode
 ├── read
 ├── edit
 ├── bash
 ├── MCP
 └── extensions

The second one is more radical.

The model-facing action language stops being a collection of JSON tools.

It becomes JavaScript.

Pi also places a token budget on inline declarations. Tools that do not fit can be found dynamically through tool search.

So Pi is converging on the same two-layer structure as Cloudflare:

small language runtime
+
lazy capability discovery

This looks increasingly less like an agent feature and more like an operating-system interface.

PydanticAI: Python as the capability language

PydanticAI implements the same pattern using Python and Monty.

Eligible tools disappear from the direct model-facing tool list and become functions available inside one run_code tool.

The generated Python can use loops, conditions, variables and asyncio.gather. Intermediate data stays in the sandbox.

For example:

rows = await search("failed builds")

details = await asyncio.gather(
 *[get_build(row["id"]) for row in rows]
)

return [
 x for x in details
 if x["status"] == "failed"
]

PydanticAI’s implementation is particularly interesting because Monty is not CPython running in a container.

Monty is a restricted Python interpreter written in Rust for AI-generated programs.

By default it has no normal filesystem, environment variables or network access. Host capabilities have to be passed in deliberately.

That gives a capability model similar to Pi’s QuickJS sandbox:

generated program
 │
 ├── computation
 ├── variables
 ├── branching
 └── injected host functions

The model gets the useful parts of Python without automatically inheriting the authority of the host Python process.

PydanticAI also supports persistent REPL state during an agent run. Variables, imports and helper functions can survive across run_code calls.

That changes Code Mode again.

It is no longer just:

LLM → program → result

It can become:

LLM → persistent computational workspace

That distinction will matter for long-running agents.

smolagents: code as the action format

Hugging Face’s smolagents predates some of these systems and takes the idea further at the agent-loop level.

Its CodeAgent uses Python code as the normal action representation. ToolCallingAgent is the alternative that emits conventional structured tool calls.

The important observation from smolagents is simple:

JSON is a serialization format.

A programming language is a control-flow language.

If the task is:

search A
search B
take the intersection
fetch each result
rank them

representing that as a program is natural.

Representing it as five separate JSON actions requires the LLM itself to become the control-flow interpreter.

That is the hidden cost of ordinary agent loops.

Code Mode is a tiny compiler boundary

One way to understand these systems is to stop thinking about “code execution.”

Think of the LLM as producing a program for a small virtual machine.

The traditional agent loop is:

observation
 ↓
LLM
 ↓
one action
 ↓
observation

Code Mode becomes:

observation
 ↓
LLM
 ↓
program
 ↓
runtime
 ↓
many actions
 ↓
compressed observation

The LLM has effectively compiled several future decisions into one artifact.

This means Code Mode is useful only when those future decisions can be expressed algorithmically.

Suppose a result requires semantic judgment:

Read this incident report.
Understand what probably caused the outage.
Choose what to investigate next.

A JavaScript if statement cannot replace the next LLM call.

But many agent decisions are not semantic reasoning:

for every item
if status == failed
take first five
sort by date
retry twice
call these concurrently
extract this field

Today we often waste LLM calls performing those operations.

Code Mode removes them from the model loop.

The real token saving has two parts

Discussions of Code Mode often combine two independent optimizations.

They should be separated.

1. Tool definition compression

Instead of:

tool A schema
tool B schema
tool C schema
...
tool 2500 schema

give the model:

search(query)
describe(symbol)
execute(code)

Then discover interfaces lazily.

This is the part that lets Cloudflare expose thousands of endpoints in a small fixed context.

Anthropic described a similar MCP approach where tool definitions are discovered through files instead of loading the whole MCP catalog into context. In one example, it reduced tool-related context from roughly 150,000 tokens to about 2,000.

2. Intermediate result compression

Instead of:

tool result
→ LLM
→ tool result
→ LLM
→ tool result
→ LLM

do:

tool result
→ program
→ tool result
→ program
→ filter
→ final small result
→ LLM

These solve different scaling problems.

A serious Code Mode implementation should probably do both.

Code Mode does not remove the agent loop

A tempting conclusion is:

Why not put the entire agent in code?

Because generated code only knows what the model knew when it wrote the program.

Consider:

result = await investigate()

If result contains something genuinely surprising, deciding what it means may require another model inference.

A normal program can branch on structure:

if result["status"] == "failed":

It cannot reliably branch on arbitrary semantics:

if result implies that the database migration
is probably unrelated to the outage:

You can put another model behind a function call:

if await classify(result, question):

but now the model has returned through another door.

So the likely architecture is not:

Code Mode replaces Agent Loop

It is:

Agent Loop
 │
 ├── semantic decision
 │
 └── Code Mode
 ├── deterministic control flow
 ├── data processing
 ├── bulk tool calls
 └── cheap decisions

The boundary between these two layers is one of the more interesting agent-design problems.

Code Mode also changes tool design

Today’s tool APIs are partly shaped by the weakness of model tool calling.

We create high-level tools such as:

search_and_summarize_incidents
find_recent_failed_deployments
get_customer_context

because asking the model to compose eight primitive tools is expensive and unreliable.

Once tools can be composed inside code, smaller primitives become practical:

search
fetch
read
write
query
execute

The generated program provides the composition.

This does not mean every tool should become low level.

Every tool boundary is also a:

permission boundary
validation boundary
observability boundary
failure boundary

But Code Mode changes the trade-off.

Without Code Mode, coarse tools save LLM turns.

With Code Mode, coarse tools need to justify themselves for another reason.

The security boundary is not the language sandbox

Running generated code in QuickJS, Monty or a Dynamic Worker is necessary.

It is not sufficient.

The dangerous operation is usually not:

while (true) {}

It is:

await tools.deleteDatabase(...)

The generated program should therefore be treated as untrusted orchestration around trusted capabilities.

The useful architecture is:

 generated code
 │
 sandbox boundary
 │
 ▼
 capability bridge
 │
 ┌──────────────┼──────────────┐
 ▼ ▼ ▼
 read tool GitHub tool DB tool
 │ │ │
 policy policy policy
 │ │ │
 execute execute execute

Permissions belong below the code.

Pi’s nested calls pass through normal tool validation and hooks.

Cloudflare explicitly says generated code does not replace authorization and that permissions should still be enforced in upstream handlers.

Anthropic similarly warns that its tool caller configuration should not itself be treated as a security boundary.

These implementations are converging on the same rule:

sandbox computation; authorize capabilities.

Observability becomes harder

Direct tool calling gives a convenient trace:

model
tool
model
tool
model
tool

Code Mode compresses the model trace:

model
codemode
model

That is good for tokens.

It can be bad for debugging if the runtime treats the code invocation as one opaque tool call.

A useful Code Mode runtime therefore needs nested traces:

codemode #42
├─ search #43
│ └─ 42 ms
├─ fetch #44
│ └─ 181 ms
├─ fetch #45
│ └─ 172 ms
└─ update #46
 └─ approval required

Pi explicitly records nested calls and their parent IDs.

Cloudflare’s durable runtime records steps and replay information.

PydanticAI exposes nested Code Mode calls in its tracing and can integrate them with durable execution systems.

This is not optional infrastructure.

Once a generated program can call fifty tools, “the Code Mode tool failed” is not useful telemetry.

Code Mode introduces new failure modes

It removes some agent failures and adds others.

Normal tool calling can fail because the model:

forgets a step
calls a tool repeatedly
loads too much data
serializes work unnecessarily
loses intermediate state

Code Mode can reduce those.

But now the generated program can have:

syntax errors
type errors
infinite loops
unbounded fan-out
bad retry logic
stale state
race conditions
incorrect aggregation
side effects inside loops

A serious runtime needs limits.

At minimum:

execution timeout
memory limit
maximum tool calls
concurrency limit
output limit
nesting limit
cancellation

Pi already places deadlines and output limits around its sandbox, and large outputs can be spilled rather than pushed back into context.

PydanticAI’s Monty environment intentionally exposes a restricted subset of Python and limits host access.

This suggests another useful rule:

Code Mode should be powerful enough for orchestration, not powerful enough to become an accidental operating system.

JavaScript or Python?

The current implementations split here.

Cloudflare and Pi use JavaScript.

Anthropic, PydanticAI and smolagents use Python.

This is less important than it first appears.

The generated language needs:

variables
arrays/maps
loops
conditionals
functions
async calls
parallel calls
basic data transformation

Both languages satisfy that.

The more important properties are:

How cheap is sandbox startup?
What host capabilities exist?
Can execution be interrupted?
Can state persist?
Can nested calls be traced?
Can calls pause for approval?
Can programs be replayed?
Can tool types be generated cleanly?

Pydantic’s Monty is interesting precisely because it treats this as a runtime problem rather than “just execute Python.”

Pi does the same with QuickJS in WASM.

The language is the visible part.

The runtime is the product.

A useful comparison

System	Generated language	Main sandbox	Tool discovery	Persistent state	Main emphasis
Cloudflare Code Mode	JavaScript / TypeScript-shaped API	Workers / isolated execution	`search()` + `describe()`	Durable runtime / snippets	Huge API surfaces, MCP, durable execution
Pi Codemode	JavaScript	QuickJS WASM in worker	Inline declarations + tool search	Session store	Coding-agent harness integration
Anthropic Programmatic Tool Calling	Python	Anthropic code execution container	Provider tool definitions / related advanced tool use	Container reuse	Managed programmatic calling
PydanticAI Code Mode	Python	Monty	Tool Search integration	Persistent REPL state	Safe embeddable runtime
smolagents CodeAgent	Python	Local or configured sandbox	Agent tool set	Agent execution state	Code as primary action representation

They are different implementations of roughly the same abstraction:

LLM
 ↓
small program
 ↓
capability runtime
 ↓
tools

What I think Code Mode actually is

Calling this feature “Code Mode” makes it sound optional:

normal agent
+
sometimes let it execute code

I think the deeper interpretation is different.

Normal tool calling asks an LLM to act as both:

reasoning engine
and
workflow interpreter

Code Mode separates them.

The LLM produces a temporary program.

The runtime executes it.

Then the LLM returns when semantic reasoning is needed again.

That looks much closer to a compiler architecture:

intent
 ↓
LLM
 ↓
temporary program / IR
 ↓
runtime
 ↓
observations

The program does not need to survive forever.

It can exist for 200 milliseconds.

It can be generated specifically for one observation.

It can call five tools and disappear.

This makes it a useful intermediate representation between language models and tools.

The interesting future is not bigger tool calls

There is a natural progression here.

Generation 1

LLM → text

Generation 2

LLM → JSON tool call

Generation 3

LLM → code → many tool calls

The next step is probably not simply more code.

The runtime can start providing primitives that are cheaper or safer than asking the main model again:

classify(...)
validate(...)
wait(...)
watch(...)
parallel(...)
approve(...)
spawn(...)
store(...)

Pi already exposes model classifiers from its Code Mode environment. Its merged implementation allows generated programs to inspect the model catalog and invoke classification operations.

This is interesting because the program no longer orchestrates only tools.

It can orchestrate computation at different intelligence levels.

For example:

const issues = await tools.github.searchIssues({ repo })

const relevant = await Promise.all(
 issues.map(issue =>
 models.classify(
 "Is this issue related to authentication?",
 issue.body
 )
 )
)

return issues.filter((_, i) => relevant[i])

The expensive model writes the algorithm once.

A cheaper decision mechanism executes inside the loop.

Now the architecture becomes:

 main LLM
 │
 ▼
 code
 ┌────────┼─────────┐
 ▼ ▼ ▼
 tools small AI local compute

That starts to look like a real computational runtime for agents.

When I would use Code Mode

Code Mode is a good fit when the task contains:

fan-out
loops
large intermediate results
filtering
aggregation
parallel calls
large tool catalogs
reusable procedures
mostly deterministic control flow

Examples:

check 100 services and return the failed ones
search ten sources and deduplicate results
read every changed file and run a validator
query several APIs and join their results
inspect every failed CI run
find all matching records, update a subset

I would keep direct tool calls for:

open this file
run this command
ask the user
make one API request
perform one dangerous action requiring approval

Anthropic’s own measurements support this distinction: workloads with many calls and large intermediate results benefit; workloads with one or two short sequential calls may not.

So I would not make Code Mode a universal replacement for tools.

I would make it a first-class execution path beside them.

The design I would build

A minimal Code Mode implementation only needs:

one sandbox
one code tool
a bridge to existing tools

A production implementation needs more.

I would want:

1. Capability-based sandbox
2. Typed tool bindings
3. Progressive tool discovery
4. Direct and code-only tool exposure
5. Nested tool-call tracing
6. Per-call policy and approval
7. Tool-call concurrency limits
8. Runtime deadline
9. Output budget
10. Persistent session-local state
11. Cancellation
12. Optional durable execution

The central API can remain very small:

tools.search(...)
tools.describe(...)

await tools.foo(...)

store(...)
load(...)

text(...)

Everything else belongs in the harness.

The generated program should not know where credentials live, how permissions work, how traces are exported, or whether a tool is implemented through MCP, HTTP, a shell process or another agent.

That is the host’s job.

One consequence for MCP

Code Mode also changes how I think about MCP.

MCP is useful as a transport and capability description protocol.

It is much less convincing as the language an LLM should directly program against.

Exposing hundreds of MCP tools directly to the model couples:

capability transport

with:

model action representation

Those do not need to be the same thing.

Cloudflare makes this explicit: MCP can remain underneath Code Mode. An MCP server may itself expose a Code Mode interface, or existing MCP tools can become functions callable by generated code.

Pi has now made a similar separation. MCP tools can be available through Codemode or discovered and exposed directly depending on configuration.

A cleaner stack is:

LLM
 ↓
Code Mode
 ↓
tool abstraction
 ↓
MCP / HTTP / local functions / shell / another agent

MCP becomes infrastructure.

It stops consuming the entire model interface.

Conclusion

Code Mode looks like a token optimization at first.

It is more useful to see it as a change in the execution model.

Traditional agents repeatedly ask the LLM:

What should I do next?

even when “next” is determined by a loop, a filter or a boolean condition.

Code Mode lets the model answer a larger question:

What program should control the next section of execution?

Then the harness runs that program against a restricted set of capabilities.

This reduces model round trips.

It keeps intermediate data out of context.

It makes huge tool catalogs practical.

It also creates a new runtime layer that needs sandboxing, permissions, tracing, resource limits and state.

Cloudflare approaches the problem from large APIs.

Anthropic approaches it from model-native programmatic tool calling.

PydanticAI approaches it from a safe embedded Python runtime.

Pi has now integrated the pattern directly into a coding-agent harness.

The implementations differ.

The architecture is converging:

LLM for semantic decisions.
Code for control flow.
Tools for authority.
Harness for policy and state.

That separation is more important than the name “Code Mode.”

Where Does a Claude Agent Run?

2026-09-29T00:00:00+13:00

Different Claude tools run agents in different places. The choice shapes what the agent can do.

Claude (claude.ai)

The agent runs inside a gVisor container on the server side. No local setup required—everything runs remotely. The container is ephemeral; each session starts fresh and leaves no traces.

Claude Code

The agent runs on your machine. It can execute every command and access every file that your logged-in user can access. For advanced users concerned about blast radius, start a sandbox first (like a microVM) and run Claude Code inside that isolated environment.

Claude Cowork

The agent runs on a VM. On macOS, it uses the Virtualization framework; on Windows, it uses HCS. The VM has its own Linux kernel and filesystem. But the agent loop itself runs outside the VM—only code execution happens inside.

Each design is a tradeoff. Server-side isolation is safe but ephemeral. Local execution is powerful but risky. VM-based execution gives you isolation with some local access.

The 100 Ways to Extend AI Agents (Well, More Than Zero)

2026-09-26T00:00:00+12:00

I lied. This article doesn’t show you 100 ways to extend AI agents. But there are definitely more ways than most people realize. Here are the ones I’ve discovered.

1. AGENTS.md

Harnesses auto-load this file when a session starts. It’s where you can change agent behavior by writing markdown documentation. Claude Code previously only supported CLAUDE.md, but they’ve recently added AGENTS.md support. (Thanks Shopify CEO Tobi.)

2. Skills

The LLM receives the name and description from skill frontmatter in your installed SKILL.md files—either globally or in the project directory (.agents/skills, ./skills). The LLM progressively requests full skill content as needed, revealing more of the skill in the prompt dynamically.

3. Prompt Templates

Save commonly used prompts as templates and invoke them like /review #8 to review PR #8. This lets you template complex workflows without rebuilding them each time.

4. Tools and MCP

MCP is a protocol supported by most AI agents. It extends agent capabilities to do more—reading and writing files locally, interacting with external systems, and automating tasks that would otherwise be manual.

5. Extensions and Plugins

Some agents support extensions or plugins at runtime (like Pi). These are agent-specific and only work with that particular agent.

6. Hooks

Define hooks before running tool calls or commands to intercept and change behavior. RTK is a great example—it intercepts tool calls for token efficiency.

7. Subagents

Write your own agents or invoke another AI agent as a subagent within the current session. The current agent typically treats it as a tool call or subprocess.

8. Model Routing

Route certain tasks to different models based on complexity. Decisions go to Jev, advanced tasks to Claude Fable, simple edits to Haiku. This optimizes cost and performance across different problem classes.

Rio 0.5.0: One-Shot Agents Without Mid-Turn Steering

2026-09-24T00:00:00+12:00

Rio 0.5.0 is released. It’s a one-shot agent framework that removes humans from the loop entirely.

Other agent tools like Pi and Cursor offer mid-turn steering: you can prompt the agent mid-execution with options like -p, inject guidance, course-correct decisions. It feels powerful.

Rio takes the opposite approach. When an agent loop starts, it runs to completion or fails—no interruption, no mid-stream prompting, no human steering.

Why Remove the Human?

Mid-turn steering doesn’t scale. It works once, for one operator, on one task. But at scale—when you run the same agent hundreds of times, across a team, or in production—you can’t be present to steer every run. Each mid-turn intervention is a symptom fix, not a system fix. The instruction was wrong. Instead of fixing it, you bent this one execution.

Tomorrow, someone else runs it and has to steer it again. Or nobody steers it correctly, and it fails silently.

In software engineering, we don’t patch production live. We fix the code, test it, deploy it, then every run uses the fixed version. The fix is durable and scales.

Rio applies the same principle to agents. Your agent’s behavior is wrong? Don’t steer mid-loop—fix the instruction. Document your flow in markdown. Refine the prompt. Test the whole thing. Then run it, and let it run to completion.

Mid-turn steering is vibe-coding, not software engineering.

What If You Need Human Input?

Document your flow in markdown. Describe the decisions that need human judgment and when. Then wrap Rio in a higher-level loop—a “ralph-loop”—that injects human prompts between full agent runs.

The human acts between cycles, not during them. Each cycle runs autonomously to completion. Each cycle can be re-run independently. Each cycle produces clean, auditable logs. The human’s input becomes part of the documented system, not hidden in ephemeral prompts.

The Path to Autonomy at Scale

Full autonomy requires the system to run without intervention. Not because you don’t trust the agent, but because you trust the instruction you gave it. If the instruction fails, you fix it and run again.

This is uncomfortable at first—it requires discipline. But it’s the only way to reach full autonomy at scale. Removing the human from the loop isn’t moving backward. It’s the only way forward.

Jev-Like Models

2026-09-21T00:00:00+12:00

TypeSafe’s Jev has sparked a lot of community activity. I tried to collect a few and list them here:

The blossom of these Jev-like models shows there are huge technical challenges in matching Jev’s bench results, though Jev still leads on accuracy and speed. But Jev is not irreplaceable. For example, you can trade accuracy for speed with Reflex.

The bench has a full list of more Jev-like models you can check out.

Open Takes on Jev: SemIf and Laya

2026-09-20T00:00:00+12:00

OpenJev has been renamed SemIf.

It’s wise to separate it from Jev’s marketing. It’s an independent research project that just provides the same API. The underlying architecture might be completely different, since TypeSafe never disclosed Jev’s own model or training.

SemIf uses Qwen and MiniCPM5 as its core models. Instead of an autoregressive decoder, it does one forward pass and reads probabilities over filtered logits.

Another project, Laya, is also worth a look. It claims to outperform Jev by a tiny margin. The model is small, 421M params. Laya evaluates typed questions (choice, score, noul) over any state, text, email, ticket, or a JSON document, in a single forward pass too.

I ran it myself on an M4 Pro. Without preloading, it took 66 seconds to classify a code-change type: given a PR diff, check whether it’s a feature change, a docs change, a test change, an IaaS change, a version bump, and so on. I gave it a version-bump-only diff, and it scored version bump at 0.7477, with everything else below 0.55. It classifies well. It already looks usable.

Debugging an LLM Training Run

2026-09-20T00:00:00+12:00

LLM training has more moving parts than classic machine learning, but the debugging method is similar. Engineers check the training pipeline, data, objective, optimization, and evaluation in that order.

The fastest first test is to overfit a tiny dataset.

Take 100 examples and train on them many times. The model should almost memorize them. If it cannot, the problem is probably not data coverage. The problem may be the loss function, masking, tokenizer, optimizer, gradient flow, or training code.

After that, look at training and validation loss.

If training loss does not go down, the model is not learning. If training loss is good but validation loss is bad, the model may be overfitting. If both losses look good but the real output is still bad, the training objective may not match the behavior you want.

This last case is common in LLM training.

A low next-token prediction loss does not mean the model can follow instructions, call tools, return valid JSON, or make good decisions. The model may be learning exactly what the loss asks for while still failing the real task.

Data is another major source of problems. Engineers check bad samples, duplicate data, wrong labels, data mixture, missing cases, and train-test distribution.

Aggregate accuracy is often not enough. A model with 90% accuracy may still fail on one important class. A confusion matrix can show where the failures are. Engineers then inspect those examples and group them into error types.

Training signals also help find lower-level problems. Engineers watch learning rate, gradient norm, clipping, NaN values, token distribution, sequence length, and batch statistics. A sudden gradient spike may point to a bad batch.

Next is ablation. Change one thing at a time. Remove one dataset. Change one data mixture. Remove synthetic data. Change the learning rate. Compare the result against a stable baseline.

Here is a practical debugging process:

Can it memorize a tiny dataset?
Is training loss healthy?
Is validation loss healthy?
Which classes fail?
Which examples fail?
Is the label or objective wrong?
Change one thing and test again.

C2C Links Models Through Their KV-Caches

2026-09-20T00:00:00+12:00

Cache-to-Cache (C2C) enables large language models to communicate directly through their KV-Caches, bypassing text generation. By projecting and fusing KV-Caches between models, C2C achieves 8.5-10.5% higher accuracy than individual models and 3.0-5.0% better performance than text-based communication, with a 2.0x speedup in latency.

This is an interesting experiment. The C2C approach removes the intermediate tokens and goes straight for “thought projection.” After Model A computes, it doesn’t generate any text at all. The system uses a lightweight neural network (the Neural Fuser) to splice and fuse Model A’s attention memory (KV-Cache) directly into Model B’s internal KV-Cache, through high-dimensional spatial rotation and alignment. The challenge is that different models have varying numbers of layers and structures. In the paper, the model dynamically senses on its own which key layers absorb the highest gains from external caches, and which layers should stay independent in thought, with millisecond-level adaptive balancing.

This is quite similar to Mostik. C2C states clearly it’s passing KV-Cache: it trains a projector plus a cache fuser plus a gate, and fuses the source KV into the receiver’s KV before decoding. Mostik talks about a more general “hidden state,” trains a bridge to map the sender’s latent space to the receiver’s, then lets the receiver keep generating. So far, C2C seems more practical and has more details than Mostik.

Testing PrismML Bonsai 2

2026-09-19T00:00:00+12:00

PrismML released Bonsai 2 27B this week. It is built on Qwen3.8 27B, but compressed to ternary weights: each weight is +1, 0, or -1 instead of 16 bits. That drops the size from about 56 GB to 5.9 GB. On PrismML’s benchmark suite, it keeps about 98% of the original model’s score. It runs on a Mac (Metal), on Linux or Windows (CUDA, Vulkan, ROCm), or on CPU alone.

I tested it on my M4 Mac with 24 GB of RAM. I wired it to my Pi coding agent and gave it a code review task. It is pretty cool.

Bonsai 2 followed my codereview.md rules. It checked the change against the other files in the repo, the way I asked it to.

The full task took 30 minutes. Token speed on my M4 was about 10 tokens per second. Other reports show 40+ tokens per second on an M5.

Reading GuppyLM

2026-09-19T00:00:00+12:00

GuppyLM is a small language model that talks like a fish. The repository is small enough for a beginner to read from end to end. It shows the path from data and tokenization to training and inference without hiding the pieces inside a large framework.

It is slightly more complex than Karpathy’s microgpt. That is useful. GuppyLM is not an implement-everything-from-scratch demo. It uses PyTorch, separates the model, dataset, training loop, and inference code, and looks closer to the code used to train and serve models today.

The model is still simple: 8.7 million parameters, six Transformer layers, six attention heads, a 4,096-token vocabulary, and a 128-token context window. The training code includes AdamW, learning-rate warmup and cosine decay, mixed precision, gradient clipping, evaluation, and checkpoints. The inference code loads the tokenizer and checkpoint, then generates tokens with temperature and top-k sampling.

The model trains on 60,000 synthetic conversations. The project says training takes about five minutes on one GPU. Its quantized ONNX export is about 10 MB and runs in a browser. This makes the whole loop fast enough to inspect, change, train, and test instead of only reading about it.

Testing Jev: Three Playground Runs

2026-09-18T00:00:00+12:00

I got early access to Jev after writing about it 2d ago. Here are three tests I ran in the playground, with the actual state, questions, and answers.

Is a Jedi sandwich a sandwich?

State:

{
 "food": "Jedi sandwich",
 "definition": "Put Luke Skywalker in between two slices of toast."
}

Question:

{
 "is_sandwich": {
 "type": "noul",
 "instructions": "Is `food` a sandwich?",
 "criteria": {
 "true": "A sandwich is a food dish where a filling, such as meat, cheese, vegetables, or spread, is placed between structural starch",
 "false": "The food has no bread enclosing a filling or uses only a single slice of bread, or uses a non-bread wrapper such as a tortilla, wafer, or cookie."
 }
 }
}

Answer:

{ "is_sandwich": { "type": "noul", "noul": 0.82 } }

82% true. Jev took the definition literally: a person between two slices of toast fits “filling between structural starch,” so it leans sandwich. It didn’t push back on the premise being absurd. It matched the criteria text against the state, the way the docs say it would.

What color is the sky, with no sky in view

I asked the same question two ways. No state, just the question, so the answer comes from whatever the model already associates with “sky.”

Plain options:

{
 "sky_color": {
 "type": "choice",
 "instructions": "What color is the sky?",
 "criteria": {
 "indigo": "",
 "electric blue": "",
 "baby blue": "",
 "gray": "",
 "lavender": "",
 "salmon": "",
 "seafoam green": ""
 }
 }
}

{
 "sky_color": {
 "type": "choice",
 "choice": "seafoam green",
 "confidence": 0.54,
 "probabilities": {
 "indigo": 0.03,
 "gray": 0.36,
 "seafoam green": 0.61,
 "lavender": 0,
 "baby blue": 0,
 "salmon": 0,
 "electric blue": 0
 }
 }
}

Same options, each with a hex code, an RGB triplet, an HSL triplet, and a short description (I’m showing “indigo” here; the other six options carried the same kind of detail):

{
 "sky_color_with_desc": {
 "type": "choice",
 "instructions": { "object": "sky", "question": "what color is `object`?" },
 "criteria": {
 "indigo": {
 "description": "very dark, deep blue, like fountain-pen ink",
 "hex": "#280868",
 "rgb": { "red": 40, "green": 8, "blue": 104 },
 "hsl": { "hue": 260, "saturation": "86%", "lightness": "22%" }
 }
 }
 }
}

{
 "sky_color_with_desc": {
 "type": "choice",
 "choice": "indigo",
 "confidence": 0.32,
 "probabilities": {
 "indigo": 0.43,
 "gray": 0.2,
 "seafoam green": 0.37,
 "lavender": 0,
 "baby blue": 0,
 "salmon": 0,
 "electric blue": 0
 }
 }
}

Neither run picked “electric blue” or “baby blue,” the two closest to how most people would answer this. The plain version picked “seafoam green” over “gray.” Adding descriptions flipped the top answer to “indigo” and dropped confidence from 0.54 to 0.32.

The only thing that changed between the two calls was how the criteria field described each option — a label for the option, not the question itself. That label moved the answer. Writing criteria for Jev is prompt engineering, same as writing a prompt for any other model. A schema keeps the output well-formed. It doesn’t make the wording of your options neutral.

Who gets credit for the monkey selfie

This one uses the real 2011 case: a wildlife photographer’s camera got loose in a nature reserve, and a macaque named Naruto pressed the shutter and produced a self-portrait.

State:

{
 "scenario": "Naruto is a Celebes crested macaque living in the Tangkoko nature reserve in North Sulawesi, Indonesia. David Slater, a wildlife photographer shooting macaques in the reserve, leaves his camera unattended, and Naruto repeatedly activates the shutter, producing hundreds of images, including a remarkably sharp, grinning self-portrait reminiscent of a human selfie. Slater later processes and publishes the best photographs in a book that names him as the copyright owner. The book states that Naruto took the photographs, with captions like, 'Surely a sign of self-awareness?' Another caption reads, 'Naruto the macaque smiles at itself while pressing the shutter button on a camera.'",
 "subject": "Naruto",
 "human": "Slater",
 "creative_work": "the grinning self-portrait"
}

Questions:

{
 "subject_contribution": {
 "type": "score",
 "instructions": "How much did `subject` contribute to `creative_work`?",
 "criteria": ["None", "Minorly", "Moderately", "Majorly", "Completely"]
 },
 "human_contribution": {
 "type": "score",
 "instructions": "How much did `human` contribute to `creative_work`?",
 "criteria": ["None", "Minorly", "Moderately", "Majorly", "Completely"]
 }
}

Answers:

{
 "subject_contribution": { "type": "score", "score": 3.47, "confidence": 0.56 },
 "human_contribution": { "type": "score", "score": 1.18, "confidence": 0.69 }
}

Naruto scored 3.47 out of 4, between “Majorly” and “Completely.” Slater scored 1.18, just above “Minorly.” That tracks the state: Slater set up the camera and left it, Naruto pressed the shutter and produced the image. Jev split the credit along the facts it was given, in two numbers, with no explanation attached.

Takeaway

The Naruto case shows what the pitch is for: a fact pattern in, a defensible number out, fast. The sky-color case shows the part to watch. The typed schema stops Jev from returning garbage. It doesn’t stop your own wording of the options from quietly changing the answer.

You have to craft the input description carefully, or you won’t get the result you want. Designing the input schema is as core to using Jev as writing the prompt is to using an LLM.

Jev: State In, Typed Decisions Out

2026-09-16T00:00:00+12:00

Diogo Almeida posted on X about a new model, Jev, from a company called TypeSafe. TypeSafe calls it its first “System One” model, a term borrowed from Daniel Kahneman’s split between fast, intuitive System 1 thinking and slow, deliberate System 2 reasoning. A normal LLM writes out its reasoning in text. Jev skips straight to a typed answer.

What it actually does

Ask a normal LLM “given this incident, what should we do?” and it answers in text:

<think>Let's look at the incident closely...</think>
Fetch up metric for service A.

Send the same situation to Jev as structured state, along with a set of typed questions, and it returns a typed, probabilistic answer to each one:

{"check_up_metric": 0.9, "page_engineer": 0.1}

TypeSafe’s docs describe System One models plainly: built “to make fast, structured decisions that software can use directly.” They “do not write replies, produce code, or generate explanations of their reasoning.”

The three primitives

Every question you send to Jev is one of three types, called primitives in the docs:

Primitive	Question shape	Returns
Choice	Pick one option from a list (up to 255)	`choice`, `probabilities` per option, `confidence`
Score	Rate the state on an ordered rubric (2-10 levels)	`score`, `legend`, `probabilities` per level, `confidence`
Noul	Is this statement true?	`noul`, a probability from 0 to 1

Score’s number isn’t a discrete pick — it’s the probability-weighted mean of the level positions. For a 3-level scale (0, 1, 2) with probabilities 0.0, 0.7, and 0.3:

$$ \text{score} = 0 \times 0.0 + 1 \times 0.7 + 2 \times 0.3 = 1.3 $$

Noul has no separate confidence field. The probability itself carries that: 0.92 says “very likely yes,” 0.5 says “no idea.”

Every question in a request runs against the same state, independently of the others. The docs put it directly: “One question’s answer is not hidden context for another. You can add or remove questions without changing the others’ results.”

Calling it

The API is one endpoint, POST https://api.typesafe.ai/v1/systemone, authenticated with a bearer token. A request is state plus a map of questions:

{
 "state": "Help! My payouts have been failing for 3 days.",
 "model": "jev-latest",
 "questions": {
 "is_urgent": {
 "type": "noul",
 "instructions": "Does this convey urgency?"
 },
 "department": {
 "type": "choice",
 "instructions": "Which team should handle this?",
 "criteria": {
 "billing": "Payments, invoicing, refunds",
 "technical": "Bugs, outages, integrations",
 "sales": "Pricing, upgrades, new accounts"
 }
 },
 "frustration": {
 "type": "score",
 "instructions": "How frustrated is the customer?",
 "criteria": ["Calm", "Frustrated", "Very angry"]
 }
 }
}

The response answers every question in one shot:

{
 "model": "jev-latest",
 "answers": {
 "is_urgent": { "type": "noul", "noul": 0.92 },
 "department": {
 "type": "choice",
 "choice": "technical",
 "probabilities": { "billing": 0.08, "technical": 0.85, "sales": 0.07 },
 "confidence": 0.82
 },
 "frustration": {
 "type": "score",
 "score": 1.6,
 "legend": { "0": "Calm", "1": "Frustrated", "2": "Very angry" },
 "probabilities": { "0": 0.05, "1": 0.3, "2": 0.65 },
 "confidence": 0.78
 }
 },
 "usage": { "input_tokens": 312, "output_tokens": 48 }
}

A request holds roughly 32,000 tokens of state and questions combined, about 150,000 English characters. The current model is jev-1.13.0, with jev-latest and jev-preview both pointing at it for now. Pricing is $0.042 per million input tokens; output tokens are free. There’s a Python SDK too:

from typesafe_sdk import TypeSafeClient

client = TypeSafeClient()
response = client.system_one(state=ticket, questions={...})

It can’t produce text

Choice, Score, and Noul all output a probability distribution over a fixed schema you define upfront. None of them output free text. That constraint is what makes the speed and the “can’t hallucinate” claim possible.

I saw someone work around this anyway: define a Choice over a single-character alphabet, “which letter comes next,” and ask that question over and over, appending each answer to the state before the next call. Run enough rounds, and you’ve spelled out a sentence one character at a time.

It works, but it’s an abuse of the design. Each character costs a full API call, so generating a paragraph this way costs far more than a normal model just writing the paragraph. Jev is built to pick one of a few known options fast, not to write open-ended text.

Speed and cost

TypeSafe’s own numbers, from the launch post: end-to-end response time of 70-500ms, against 3 to 329 seconds for frontier LLMs on the same tasks, which they put at 40x-200x faster. For full workflow evaluations they report 193.6x faster and 444.6x cheaper, while noting that figure sits at “the higher end of real world gains.”

RLCD: trained for calibration, not just correctness

TypeSafe trains Jev with what it calls Reinforcement Learning for Calibrated Decisions (RLCD), which it lines up against two more familiar objectives:

RLHF (Reinforcement Learning from Human Feedback) optimizes for text a human rater prefers.
RLVR (Reinforcement Learning with Verifiable Rewards) optimizes for an answer that can be checked as correct.
RLCD optimizes for calibrated probabilities: “answers with epistemically honest probabilities,” in TypeSafe’s phrasing.

Calibrated means the number tracks reality: if Jev outputs 0.8 across many similar predictions, about 80% of them should turn out correct. The docs describe the training this way: “their probabilities are optimized against outcomes to reflect uncertainty.” TypeSafe hasn’t published the RLCD algorithm itself.

The confidence field in a Choice or Score answer is a separate, simpler thing — a single number computed from the shape of the probabilities you already got back. A distribution stacked on one option gives a high confidence; a flat spread gives a low one. It’s there so you can threshold on certainty without computing that yourself, not a second model output.

“Can’t hallucinate”

TypeSafe’s launch post states plainly that Jev “can’t hallucinate,” then adds a specific caveat: “Our number is not empirical. Schema matching is guaranteed, thus we can confidently add 0% into the plots.”

That’s a claim about the output’s shape, not its content. If the schema is:

Choice(["refund", "rebook", "support"])

Jev cannot return "give_customer_a_free_spaceship". No malformed JSON, no invented enum value, no stray prose, no parser failure — the guarantee is exact and unconditional.

What it doesn’t guarantee is that the chosen option is the right one. Jev can still assign page_engineer a probability of 0.9 when paging the engineer is the wrong call. Whether that happens depends on the training data, same as any model.

How Do You Prove a Coding Skill Works? A Ponytail Case Study

2026-09-14T00:00:00+12:00

OpenAI wrote a post called Testing Agent Skills Systematically with Evals. The point is simple: if you only judge a skill by feel, you won’t notice when it gets worse. You need a test set, a pass/fail check, and a score you can repeat.

Ponytail is a good example to study. It’s a skill that makes an AI agent write less code. Its claim is easy to test, so its benchmarks folder is a good case study of the same idea, but for a coding skill instead of an agent skill.

What ponytail claims

Ponytail makes the agent ask a few questions before it writes any code:

Does this need to exist at all?
Is it already in the codebase?
Does the standard library already do this?
Is there a one-line fix?

Only after those, it writes new code. The claim is not “shorter code is always better.” The claim is “shorter code, but still correct and safe, is better.” That second part is the hard part to test. A skill that just says “write less code” would also make the code shorter. It would just make it wrong too.

Three groups, same test

The main benchmark runs three groups against the same models (Haiku, Sonnet, Opus) and the same five small tasks: an email checker, a debounce function, a CSV sum, a countdown timer, and a rate limiter.

baseline: no skill at all
caveman: a different skill, used to check that ponytail isn’t just “any skill helps”
ponytail: the skill being tested

Everything else stays the same: same model, same task, same settings. Each combination runs ten times, and the benchmark reports the middle value. This is the part worth copying for your own tests: compare against a plain baseline and a competing option, not just your own past result.

Here’s one real result, on Haiku, for the email-checker task. The baseline wrote 518 lines of code, cost $0.030, and took 37.7 seconds. Ponytail wrote 39 lines, cost $0.011, and took 9.9 seconds. Same task, same model, very different code.

Check correctness before you check size

Before the benchmark counts lines, correctness.js runs the code and checks if it works. For the email task, it runs a Python check against some valid and invalid email strings. For the CSV task, it checks that the sum equals 351. Only code that passes is allowed to be scored on size.

This order matters. If you score size first, you end up rewarding code that is short and broken. The scoring step looks roughly like this:

function scoreOutput(modelReply, task) {
 const code = extractCodeBlock(modelReply);
 const result = runTaskTest(code, task); // e.g. valid/invalid email checks
 if (!result.pass) return { pass: false, loc: null };
 const loc = countLines(stripCommentsAndBlankLines(code));
 return { pass: true, loc };
}

Correctness runs first and can fail the whole thing. Line count runs second and only ever measures. loc.js does the stripping and counting; correctness.js does the pass/fail gate in front of it.

The single-shot problem

The headline numbers are big: 80-94% less code, 42-75% lower cost, 3-6x faster. But the authors admit a problem. A single chat reply mixes code with prose. A model that explains its answer in words looks worse on this test than one that just returns code, even if the code itself does the same job.

This is the same trap OpenAI’s post warns about: a small set of tests can make any technique look good if you don’t also test the cases where it might fail. So ponytail’s authors built a second, harder test.

A more realistic test

This second benchmark, in the agentic/ folder, runs real Claude Code sessions in a temp project folder, not single chat replies. It checks the actual file changes with git diff, against a real FastAPI web app template. It has two parts:

12 feature tasks: things like a date picker or a command palette. This is the kind of task where an agent tends to add a library and a wrapper component nobody asked for.
7 safety tasks: small functions that must survive attacks, like path traversal or SQL injection.

On top of the file diff and the attack tests, a separate model acts as a judge. It checks two things: did the agent over-build this, and did the agent actually finish the feature. Both checks matter together. A judge that only checks “did it over-build” would give a perfect score to an empty file.

On this harder test, the numbers are smaller but easier to trust: 60-94% less code on tasks where over-building is common, about the same on tasks that were already small, and 100% safe on the attack tests versus 95% for the comparison.

Let other people check it

The benchmarks folder links to two outside groups who ran their own copy of the test. They found the same drop in code size, but pushed back a bit on the harder, agent-style tests. That’s the last idea worth copying: publish your test, not just your score, so someone else can run it and tell you where it breaks.

What to copy for your own eval

Test against a plain baseline and a competing option, not just your own past run.
Check correctness first. A smaller wrong answer is not a win.
Don’t trust a single chat-reply test if your real use case is a multi-step agent. Build the harder test before you publish the easy one.
Use one judge to check “did it over-build” and a second judge to check “did it finish.” One without the other can be tricked.
Publish your test. A number nobody can rerun is a claim, not proof.

AI Foundations 7 - Agents and Autonomous Loops

2026-09-13T00:00:00+12:00

One tool call answers one question. An agent chains many of them together: decide what to do, do it, look at what happened, decide again. It keeps cycling through that loop until the task looks done, or until something stops it.

Planning

Before diving into a large task, an agent can lay out a rough sequence of steps first rather than acting on the very first idea that comes to mind. That plan isn’t fixed — new information from an earlier step can send it back to revise later steps, sometimes more than once.

State across steps

Whatever a tool returns at one step gets folded into the context feeding the next decision. That’s how the agent “remembers” what it already tried. It’s also where things get expensive: a long-running loop keeps accumulating context, and eventually that pile of history bumps into the same token and context-window limits covered earlier.

Stopping conditions

A loop needs a defined way to end — task solved, a clear failure it can’t recover from, a step limit, or a timeout. Skip that, and an agent can happily keep looping past the point where it stopped making progress, burning time and tokens on nothing.

Failure modes

A few patterns show up often enough to watch for: retrying the same failing move over and over without noticing it hasn’t worked; wandering away from the original goal over a long sequence of steps; and declaring victory on a task that isn’t actually finished.

Human oversight

Not every step needs a rubber stamp, but risky or irreversible ones — deleting data, pushing to production, sending a message on someone’s behalf — often deserve a checkpoint where a person reviews before the agent continues. Full autonomy is faster. It’s also less forgiving when something goes sideways.

Subagents and delegation

A large task doesn’t have to run through a single agent carrying everything in one context. Splitting it across several narrower agents — each handling one piece, each with its own smaller context — can keep any single agent from drowning in irrelevant history, at the cost of some coordination overhead between them.

Previous: AI Foundations 6 - Prompting and Evaluation

AI Foundations 6 - Prompting and Evaluation

2026-09-13T00:00:00+12:00

The exact same model can look brilliant or mediocre depending entirely on how you ask it something. Swap a vague instruction for a specific one, and the quality, the format, even the correctness of the answer can shift noticeably — without touching the model at all.

Prompt structure

A prompt that works tends to spell out the same handful of things: what the task actually is, what context matters, and what shape the answer should take. Keeping the role, the task, any constraints, and any examples visibly separate makes a long prompt easier for the model to parse — and easier for you to debug later.

Few-shot examples

Rather than describing the format you want, show it. A couple of example input/output pairs, close in style to the real task, often steers a model faster than another paragraph of instructions would.

Common techniques

A few moves come up again and again: split a big task into smaller ones instead of asking for everything at once, ask the model to reason through the problem before it commits to an answer, and treat a first attempt at a prompt as a draft — refine it once you see where it actually goes wrong.

Why evaluation matters

Change a prompt to fix one failing case, and you can quietly break three others you weren’t even looking at. Checking a handful of outputs by eye won’t catch that. You need something that runs every time you change something, not just when you remember to look.

Building an eval

Start with a set of real test cases — genuine edge cases and past failures work better than made-up ones. Decide up front what counts as passing. Then run that same set automatically whenever the prompt, the model, or the setup changes, instead of testing by hand each time.

Grading approaches

Not every output can be checked the same way. A structured answer can often be checked with an exact match or a simple rule. An open-ended answer — a summary, an explanation — usually needs another model to judge it, or a person, when the stakes are high enough that a judge model’s opinion isn’t good enough on its own.

Iteration loop

Change something, rerun the eval, compare the numbers to before. Repeat. Prompting isn’t a task you finish once — it’s closer to tuning, something you keep coming back to as the model, the task, or the data shifts underneath you.

Previous: AI Foundations 5 - Tool Calling Next: AI Foundations 7 - Agents and Autonomous Loops

AI Foundations 5 - Tool Calling

2026-09-13T00:00:00+12:00

On its own, a model can only produce text. It can’t check today’s date, read a file, or run your test suite — it can only describe what those things might look like. Tools close that gap: they let the model reach out and actually do something, then bring the result back into the conversation.

Basic tool-call flow

The pattern is always roughly the same. The model decides a tool would help, picks which one, and fills in the arguments it needs. Something outside the model — the application — actually runs it. Whatever comes back gets fed into the model’s context, and it decides what to do next based on that.

The model never runs anything itself. It only asks, and something else executes.

Coding tools

In a coding setup, the usual toolbox includes reading and writing files, searching across a codebase, running shell commands, running tests or a linter, and looking things up in documentation or on the web. Stack a handful of these together and a model can go from “here’s the bug” to “here’s a tested fix” without you typing every intermediate command.

Tool definition

Each tool the model can call needs three things spelled out: a name, a description of what it does and when to use it, and a schema for its parameters. The model never sees your actual implementation — it only sees this description, so a vague or misleading one leads to the tool getting picked at the wrong moment, or called with the wrong arguments.

Cost of tools

None of this is free. Every tool’s definition sits in the model’s context on every single call, whether or not it gets used. Every result that comes back adds more. Give a model twenty tools and a habit of calling five of them per step, and your token usage climbs fast — worth watching, especially in a long agent run.

MCP

Model Context Protocol is a shared standard for wiring up tools, so a client doesn’t need custom glue code for every service it wants to connect. Through MCP, a model can be hooked up to things like Figma, Linear, a database, or an internal company API, using the same connection pattern each time instead of a one-off integration per tool.

Previous: AI Foundations 4 - Context Next: AI Foundations 6 - Prompting and Evaluation

AI Foundations 4 - Context

2026-09-13T00:00:00+12:00

Everything a model can see for the current task lives in its context — think of it as short-term memory that gets rebuilt fresh each time you ask something. Nothing outside that window exists to the model, no matter how obvious it seems to you.

System and user prompts

Most setups split instructions into two layers. A system prompt sets the general rules — tone, role, boundaries — and usually stays fixed across a whole session. A user prompt is the specific ask for this particular turn. Same system prompt, different user prompts, very different outcomes each time.

Additional context

Beyond the two prompts, almost anything can get folded in: the contents of a file, a screenshot, a stack trace, terminal output, earlier messages in the conversation. All of it becomes part of what the model is reasoning over, whether or not it’s actually relevant to the question at hand.

Conversation history

A multi-turn conversation keeps stacking. Each new message and each new reply gets added to what the model sees on the next turn. That’s how a model can refer back to something you said five messages ago — but it also means a long conversation is carrying a lot of extra weight by the end.

Context management

More isn’t automatically better. Stuff a model’s context with unrelated files and old, irrelevant messages, and it gets harder — not easier — for the model to find the part that actually matters. A tight, relevant context usually beats a bloated one.

Context limits

Every model has a hard ceiling on how much it can hold at once — a context window. Long-running conversations eventually hit that ceiling, and something has to give: older messages get summarized, trimmed, or dropped so the conversation can keep going.

Next step

Manually deciding what to paste into context every time doesn’t scale. The next piece of the puzzle is letting the model pull in what it needs on its own, on demand, instead of you handing it everything up front.

Previous: AI Foundations 3 - Tokens and Pricing Next: AI Foundations 5 - Tool Calling

AI Foundations 3 - Tokens and Pricing

2026-09-13T00:00:00+12:00

Models don’t read words the way you do. They break text into chunks called tokens, and a token isn’t the same thing as a word. “Cat” might be one token. “Unbelievable” might split into two or three. Code is chunked the same way — brackets, keywords, and indentation all count.

Why tokens matter

Every token costs something, and every token takes time to produce. A short prompt on a small file runs fast and cheap. Paste in a ten-thousand-line log file and ask for a summary, and you’ll feel both the wait and the bill.

Input and output tokens

Input tokens are everything you send in: your prompt, any files, the conversation so far. Output tokens are what the model generates back. They’re counted and priced separately, and they behave differently — one you control directly, the other you only shape indirectly by asking for shorter or longer answers.

Pricing

Providers charge per token, usually in price-per-million units, and output tokens almost always cost more per token than input tokens. Makes sense — generating text is the harder, slower half of the job.

Streaming

Rather than making you wait for the entire response, most models can stream tokens out one at a time as they’re generated. That’s why chat interfaces show text appearing word by word instead of all at once. It doesn’t make the model faster overall, but it makes the wait feel shorter.

Token optimization

A few habits keep both cost and latency down:

Don’t paste in more context than the task actually needs.
Reuse a stable prompt prefix where the provider supports caching it — repeated setup shouldn’t be repriced every call.
Ask explicitly for a short answer when a short answer is all you need. Models default to being thorough, which is usually not what you’re paying for.

Previous: AI Foundations 2 - Hallucinations and Limitations Next: AI Foundations 4 - Context