Engineering

How Nest checks its own work

Type checks, generated tests, a headless browser, screenshot diffs and a retry budget. A look at how Agent mode verifies a change, and where it still falls short.

Theo MarchettiCo-founder and CTO

Python code in a dark editor theme, close up on a screen
Photo: Stanislav Kondratiev / Pexels

Writing code is the easy part for a model. Knowing whether the code works is the hard part. When we started building Agent mode, now in private beta, we spent far more time on verification than on generation, because an agent that hands back broken work with confidence is worse than no agent at all.

This post describes what happens between “Nest made the change” and “Nest says it is done,” and where that process still has gaps.

The verification loop

After every set of edits, the agent runs a fixed sequence of checks. Each one is cheap compared with the one after it, so we run them in order and stop at the first failure.

  1. Type check. Every Nest app is TypeScript, and the compiler catches a large share of mistakes in a couple of seconds: a renamed field, a missing import, a function called with the wrong arguments.
  2. Build and migrate. The app is built and any database migrations run against a fresh copy of the project’s Postgres schema. A migration that would fail in production fails here first.
  3. Generated tests. The agent writes small tests for the behavior it just added, then runs them alongside any tests already in the project.
  4. Browser run. The app boots in its sandbox and a headless browser works through the affected flows.
  5. Screenshot comparison. Key screens are captured and compared with the previous version.

If everything passes, the agent writes its summary. If something fails, the output goes back to the agent as context for the next attempt.

Clicking through the app

The browser step is where most real bugs show up. Types can be correct and tests can pass while a button still does nothing because a handler was never attached.

Flows, not pages

For each change, the agent writes a short script of what a person would do: sign in as a member, open the timetable, book a class, check that the booking appears on the account page. The browser runs those steps with real clicks and real form input against the running app. Test users are created for each role, so a permissions change is checked from both sides: the admin can delete, and the member cannot.

Reading the console

During the run, we capture everything the browser and server log: uncaught exceptions, failed network requests, React warnings and slow queries. These are fed back to the agent verbatim. A message like TypeError: cannot read properties of undefined (reading 'capacity') with a stack trace is often all it needs to find the fix.

Screenshots

We compare screenshots of the changed screens before and after. The goal is not pixel-perfect matching, since many changes are meant to alter the layout. It is to catch unintended damage: a page that is suddenly blank, a modal that covers the whole screen, a section that disappeared from a page the agent was not supposed to touch. Large unexpected differences on untouched pages count as a failure.

Knowing when to stop

An agent that retries forever burns time and can drift further from what you asked for. Each run has a retry budget.

Situation What the agent does
A check fails, budget remains Reads the error, edits, reruns from the failing step
The same error appears twice Tries a different approach rather than the same fix
Budget is spent Stops, rolls back to the last passing state and reports what it tried
A decision is ambiguous Stops and asks before writing code

The last row matters as much as the others. Many failures are not technical at all. If the request does not say whether a refund should be automatic, no amount of retrying will produce the right answer. In those cases the agent asks you, with a suggested default.

Honest limits

This process catches a lot, but not everything, and we would rather say so plainly.

  • Tests come from the same model that wrote the code. If the agent misunderstands your intent, it can write a test that confirms the misunderstanding. The plan step exists partly to catch this earlier.
  • External services are simulated. Stripe runs in test mode and emails go to a capture inbox. Real-world behavior can still differ.
  • Visual quality is subjective. Screenshot checks catch breakage, not whether a layout looks good. That judgment is still yours.
  • Coverage follows the change. The agent tests what it touched and the flows next to it. A bug in a distant part of the app that happens to share a helper can slip through.

In the beta so far, most runs finish with every check passing, and the share of hand-backs that need a “that broke” follow-up is falling week by week. Until then, every agent run is saved as a version, so if something does slip past the checks, rolling back is one click away.

#agents#testing#engineering
Portrait of Theo Marchetti

Written by

Theo Marchetti

Co-founder and CTO

A row of server blades lit in blue in a data center rack

Engineering

Sandboxes that boot in 300 ms

Every Nest project runs in its own isolated sandbox. Here is how we got boot times down to about 300 ms, and why that speed matters for the agent.

Streaks of purple and green light curving through a dark frame

Product

Introducing Agent mode

Nest can now plan a change, make it across your whole project, test it in a real browser and fix what breaks before handing the app back to you.

A team cheering around laptops at a meeting table in a bright office

Company

Our $48M Series B

Nest has raised a $48M Series B led by Harbourline Ventures. Here is where we are, what the money is for and what stays the same.

Type to search every post.