You write a test for an agent. It works. You tune the agent's prompt. The test breaks.
Not because the agent is wrong. Because the test was a script — a sequence of "the customer says A, the agent should respond B, then the customer says C" — and the moment you teach the agent to ask one extra question, the sequence is wrong by one step and the test bails. The next conversation the agent has with a real human, however, is fine. The test is broken; the system is not.
You patch the test. The agent gets tuned again next week. The test breaks again.
This is the loop most teams shipping agents are in right now. Not because anyone is sloppy — because agent conversations are generated, and traditional tests assume outputs are fixed. The two assumptions don't compose. You can write a hundred scripted scenarios and they'll all rot the next time you tune the prompt that drives them.
It got to me a couple of weeks ago. We pushed a small change to make the agent think a bit more clearly during her first turn. The change worked for visitors. The test exploded. The reason: the agent now asks one additional probing question before advancing, and the script assumed she'd advance after the second answer, not the third. The test wasn't measuring the right thing. It was measuring the script.
I sat with this for a day. The honest realisation: scripted tests for an agent are like asking a film critic to evaluate a movie by counting words spoken. They don't measure whether the agent did its job. They measure whether the agent followed a particular path to do its job. Those aren't the same thing.
So we built Nouri.
Nouri is a tiny agent. She has one job: pretend to be a customer. She's given a persona — a name, a context, a goal, things she should reveal naturally and things she shouldn't volunteer — and a small list of destinations. The destinations are checkpoints, not lines of dialogue. Reach this state. Reach that one. Produce this artifact.
When Nouri talks to the agent under test, she talks like a person. She picks up cues. She adapts when the agent surprises her. She ignores filler. When the agent says "give me a moment, I'm looking that up," Nouri waits — she doesn't barrel ahead. Real customers do the same thing, mostly. The polite ones, anyway.
What the test framework actually checks isn't "did Nouri say the right thing" — Nouri can say whatever she wants. What it checks is "did the conversation arrive at the right destination." Did the agent eventually reach the right state? Did she eventually produce the artifact she was supposed to produce? Did she eventually move from intake to discovery to making something?
The dialogue is generated. The destination is fixed.
This sounds like a small detail. It's not. It's the whole shape of the thing.
Three implications fall out, the last of which is the one that's been keeping me up.
The first is resistance to test-fitting. There's an old joke in software: if you measure something, you optimise for the measure, not the thing. When the test is a script, every developer eventually learns to make the agent satisfy the script — even when the script is no longer what real customers do. With Nouri, there's nothing to fit to. Every run, she says different things. Every run, the agent has to actually solve the underlying problem. The test cannot be gamed because the test is not a fixed surface. It's a moving conversation with a goal-shaped boundary.
The second is asymmetric capability. The agent under test runs on a fairly capable model — slow, expensive, smart. Nouri runs on a much smaller, cheaper one. This is fine, and a little surprising the first time you sit with it. Why does it work? Because Nouri's job is much easier than the agent's. She only has to play human-shaped. She doesn't have to figure out anyone's business or generate creative work or maintain a coherent multi-turn strategy. She just has to be the kind of person the agent is built to help. A small model is plenty for that. The test costs less than the thing being tested. Which is the right shape for any test that's going to run on every prompt change for the rest of the project's life.
The third is the one I keep coming back to.
The reason agent products exist in the first place is that there are pieces of work humans don't have time or skill for, and a sufficiently capable model can do them. Customer service. Research. First drafts. Generative work. The bet of the whole industry is: send an agent to do the thing the human can't.
Now we're saying: the only way to actually check what the agent did is to send another agent. The work is too generative, too contextual, too adaptive for fixed rules to catch its drift. So we put a smaller, cheaper agent on the case as the inspector.
Which means the audit is now an agentic problem too. You can't escape the recursion. You build agents to do the work humans can't keep up with; you build agents to check agents because humans can't keep up with reading every conversation; eventually you'll need agents to check the checking agents, because the checking agents themselves drift. It doesn't stop at one layer. It probably doesn't stop at two.
This is uncomfortable to sit with if you wanted "agents do the boring stuff and humans do the judging." It turns out the judging is also boring at scale. Reading a thousand transcripts of a customer-facing agent is not a thing a human is going to do well. Reading those transcripts looking for specific drift — has she started repeating herself, has she stopped asking the third question, is her tone slipping — is not a thing a human is going to do consistently. Computers are good at consistency at scale. Generative computers are good at consistency at scale over generative output. Which is the only kind of computer fit for the job.
Nouri is going to grow. She's currently one persona walking one scenario. The plan is several personas walking several scenarios — the impatient one, the chatty one, the one who answers questions with three more questions, the one who refuses to provide the obvious detail, the one whose stated goal is not their real goal. Each of them lands in the test runner like a dropped pin: when she completes her scenario, the agent is healthy on that axis. When she doesn't, the agent has drifted on something specific and named.
The architecture of agent-checking-agent isn't a frontier idea. People have written about it. What's new, for us, is that it stopped being a thought experiment and became a thing in our test suite — the reason a refactor we shipped last week doesn't hang us out three months from now. It pays for itself the first time it catches something a script wouldn't have. Ours did, the second day she ran.
If you ship an agent, you'll probably end up here. Worth getting there on purpose rather than by panic.
End of Transmission.