How Nest checks its own work
Type checks, generated tests, a headless browser, screenshot diffs and a retry budget. A look at how Agent mode verifies a change, and where it still falls short.
Theo MarchettiCo-founder and CTO

Writing code is the easy part for a model. Knowing whether the code works is the hard part. When we started building Agent mode, now in private beta, we spent far more time on verification than on generation, because an agent that hands back broken work with confidence is worse than no agent at all.
This post describes what happens between “Nest made the change” and “Nest says it is done,” and where that process still has gaps.
The verification loop
After every set of edits, the agent runs a fixed sequence of checks. Each one is cheap compared with the one after it, so we run them in order and stop at the first failure.
- Type check. Every Nest app is TypeScript, and the compiler catches a large share of mistakes in a couple of seconds: a renamed field, a missing import, a function called with the wrong arguments.
- Build and migrate. The app is built and any database migrations run against a fresh copy of the project’s Postgres schema. A migration that would fail in production fails here first.
- Generated tests. The agent writes small tests for the behavior it just added, then runs them alongside any tests already in the project.
- Browser run. The app boots in its sandbox and a headless browser works through the affected flows.
- Screenshot comparison. Key screens are captured and compared with the previous version.
If everything passes, the agent writes its summary. If something fails, the output goes back to the agent as context for the next attempt.
Clicking through the app
The browser step is where most real bugs show up. Types can be correct and tests can pass while a button still does nothing because a handler was never attached.
Flows, not pages
For each change, the agent writes a short script of what a person would do: sign in as a member, open the timetable, book a class, check that the booking appears on the account page. The browser runs those steps with real clicks and real form input against the running app. Test users are created for each role, so a permissions change is checked from both sides: the admin can delete, and the member cannot.
Reading the console
During the run, we capture everything the browser and server log: uncaught exceptions, failed network requests, React warnings and slow queries. These are fed back to the agent verbatim. A message like TypeError: cannot read properties of undefined (reading 'capacity') with a stack trace is often all it needs to find the fix.
Screenshots
We compare screenshots of the changed screens before and after. The goal is not pixel-perfect matching, since many changes are meant to alter the layout. It is to catch unintended damage: a page that is suddenly blank, a modal that covers the whole screen, a section that disappeared from a page the agent was not supposed to touch. Large unexpected differences on untouched pages count as a failure.
Knowing when to stop
An agent that retries forever burns time and can drift further from what you asked for. Each run has a retry budget.
| Situation | What the agent does |
|---|---|
| A check fails, budget remains | Reads the error, edits, reruns from the failing step |
| The same error appears twice | Tries a different approach rather than the same fix |
| Budget is spent | Stops, rolls back to the last passing state and reports what it tried |
| A decision is ambiguous | Stops and asks before writing code |
The last row matters as much as the others. Many failures are not technical at all. If the request does not say whether a refund should be automatic, no amount of retrying will produce the right answer. In those cases the agent asks you, with a suggested default.
Honest limits
This process catches a lot, but not everything, and we would rather say so plainly.
- Tests come from the same model that wrote the code. If the agent misunderstands your intent, it can write a test that confirms the misunderstanding. The plan step exists partly to catch this earlier.
- External services are simulated. Stripe runs in test mode and emails go to a capture inbox. Real-world behavior can still differ.
- Visual quality is subjective. Screenshot checks catch breakage, not whether a layout looks good. That judgment is still yours.
- Coverage follows the change. The agent tests what it touched and the flows next to it. A bug in a distant part of the app that happens to share a helper can slip through.
In the beta so far, most runs finish with every check passing, and the share of hand-backs that need a “that broke” follow-up is falling week by week. Until then, every agent run is saved as a version, so if something does slip past the checks, rolling back is one click away.

Theo Marchetti
Co-founder and CTO



