Spec-driven development: criteria the agent can actually check

You hand the agent a detailed spec. It builds exactly what you described. And it's still wrong.

When the brief is clear but the finish line is not

Last time we looked at why a vague brief makes AI produce wrong code fast. But there's a second failure mode that shows up even with a careful spec. The agent finishes, declares success, and the output still misses the mark. The brief said what to build. Nobody stated clearly enough what passing looks like.

"Add input validation" is a direction. "Return a 400 status code when the email field is empty" is a destination. An agent can read both sentences. Only one of them says when the job is done.

This matters because an agent evaluating its own work needs a standard it can apply mechanically. It doesn't have your context. It doesn't know your implicit expectations. It works from the criterion you gave it. If the criterion is ambiguous, the self-check is ambiguous too. The measure is the problem.

The fix isn't to give the agent more instructions mid-task. By the time it's building, the criterion is the only signal it has about what done looks like. Changing it after the fact costs more than writing it well before the start.

A human developer fills gaps with judgment. They ask a colleague, check a similar feature, or guess from experience. That's slow, but it works. An agent fills gaps by picking the most likely interpretation of the text you gave it. When the text is specific, the pick is reliable. When it's not, you're counting on luck.

Why this overhead is growing

The gap between "here's what to build" and "here's what passing looks like" has a real cost. According to a Q3 2026 survey reported by AI Weekly, 42% of developers now say AI writes at least half their code. That's up from 12% the year before. In the same survey, 67% say they spend more time reviewing AI output than they did twelve months ago.

That's not a contradiction. It's what happens when volume goes up and the definition of "done" stays fuzzy. More code ships, more checking follows after the fact. Most of that checking is work the specification should have done before the first line ran.

The review burden feels like a quality problem, but it's often a specification problem. When a criterion says "the form should validate inputs", the output needs a human to judge whether the validation is correct. When it specifies "a red border around empty fields on submit", the output either matches that or it doesn't. One criterion produces a judgment call. The other produces a fact.

Writing tighter criteria changes what review looks like. The agent handles the mechanical check. You cover the things it can't.

What an acceptance criterion actually does

An acceptance criterion is a condition you can test. It has a clear input, a clear action, and a clear expected result. Pass or fail. No interpretation required.

Think of a student marking their own exam. If the answer sheet says "explain the water cycle", there's no way to know if three sentences are enough. If it says "name the four stages in order", the student can check the answer themselves. The criterion determines whether self-assessment is even possible. The same logic applies to an agent checking its own output: the quality of the self-check depends entirely on the quality of the criterion.

The line between a criterion and a wish is simple: can the agent check it, or can't it?

The software engineering body of knowledge, SWEBOK, covers this under Software Requirements: how requirements are found, written down, and checked. Its section on acceptance criteria explains how requirements expressed as testable conditions connect to validation. That's exactly what an agent needs: a condition it can evaluate against the output it just produced. If the condition isn't testable, the agent can't close the loop.

A criterion that can't be tested mechanically isn't a requirement yet. It's a note. And notes don't ship software.

The three things a checkable criterion names

A verifiable criterion names three things. Each one removes a decision the agent would otherwise make on your behalf. Together, they turn a description into a test.

1. The exact condition

This is the situation you're describing. Vague: "when the user logs in." Specific: "when a user submits a login request with a valid email and an incorrect password." The more specific the condition, the smaller the gap between what you meant and what the agent builds.

A vague condition like "handle errors gracefully" could mean anything from a friendly message to a silent failure. An agent filling that gap will pick something. It might pick well. It might not. Either way, it's guessing, and the guessing happens before a single line of code exists to review.

2. The input

Name what goes into the system: the type, the value, the state. "An invalid email" leaves the agent to define invalidity. "An email string with no @ character" defines it for the agent. "A large file" needs a cutoff. "A file above the maximum size limit" provides one.

Agents interpret language literally and broadly at the same time. A category label like "invalid" still needs a definition somewhere. A specific condition like "no @ character" is already the definition. One makes the agent guess. The other removes the guessing.

3. The expected output

Say exactly what the system should return, display, or store. A status code, a message string, a changed record. "Show an error" leaves the format open. "Display the message 'Please enter a valid email address' below the email field" closes it. "Reject the file" is a direction. "Return a 413 status code and do not store the file" is a result.

A September 2026 paper from arXiv makes the point directly. Open-ended AI coding dissolves verifiability and accountability. Spec-driven development is what restores them. The expected output is the part that makes a criterion verifiable rather than aspirational. Without it, the criterion describes intent. With it, the criterion describes a test.

Put the three together. Here is what the result looks like:

"Given a login form where the password field is empty, the system returns a 400 status code and displays 'Password is required' below the password field."

An agent can build to that. An agent can check that. No guesswork on either side.

BrainGrid's guide to writing criteria for AI agents describes this structure as Given-When-Then: a named condition, a named action, a named result. The name is a shorthand. The principle is that each part removes one interpretation.

How a running log of mistakes sharpens the criteria over time

Even well-written criteria have gaps the first time around. An agent finds an edge case you didn't anticipate. A term you thought was specific turns out to be ambiguous. The output passes the criterion and fails the intent. This isn't a failure of the agent. It's a signal that the criterion needed one more line.

The OWASP Top 10 for LLM Applications is a guide to common failure modes in AI-powered software. Its section on excessive agency covers what happens when an AI system has more scope than the task requires. With no precise boundaries, it has nothing to check itself against. Vague criteria are one source of that problem: if "handle errors" is the standard, the agent's permitted scope is undefined and its boundaries go unchecked.

The fix is to treat each failure as a record, not just a correction. When the agent misses the intent, write down what the criterion should have said. Note the term that was ambiguous. Note the edge case that wasn't covered. Keep that record. Consult it before writing criteria for the next task.

A simple format is enough. The task, the criterion as written, what the agent produced, and what the criterion should have said instead. You can read it back in thirty seconds before the next plan. The log doesn't need to be elaborate to be useful. It needs to be consulted.

Unlike a human developer who carries experience from one project to the next, an agent starts fresh each time. It doesn't accumulate the lessons of previous runs unless you give it access to them. The log is how you do that. It's the memory you build on purpose so the agent doesn't repeat your past mistakes.

Over time, the log becomes a checklist of the gaps your criteria tend to leave. Ambiguous terms that came up before. Edge cases you forgot to name. Input types you assumed were obvious. Each entry is a signal: the criterion needed another line.

Criteria written with that log in front of you don't repeat the same errors. The agent builds to them more reliably. The review overhead shrinks.

How this looks in a working setup

In my own projects, the shift was visible even on small tasks. An early criterion for a data export feature said "the exported file should include all required fields." The agent produced a file. It passed its own check. Several fields were missing because "required" was never defined in the criterion. The agent had no way to know what "required" meant for this context. It made a choice, and it made the wrong one.

The rewritten criterion named each field, the file format, the encoding, and what should happen if a field had no value. When the same export task ran again, the output matched. The agent didn't have to infer anything. The criterion told it.

The running log of past misses sits alongside the spec template now. Before any new criterion is written, the relevant entries are read first. The criteria are longer than they used to be. The review passes are shorter.

One sentence to carry forward

A verifiable criterion names the exact condition, the input, and the expected output. Written before any code runs, it gives the agent a definition of done it can check mechanically. A log of past misses sharpens the criteria with each cycle, so the same gaps don't reappear.

The takeaway: a criterion the agent can't test isn't a criterion yet. Writing it before code starts is the only way to find that out before it becomes a problem.

Look at your current acceptance criteria. For each one, ask: what exactly would pass this, and what exactly would fail it?

Sources

• BairesDev Q3 Barometer: 42% of Developers Say AI Writes Half or More of Their Code, Up From 12% YoY, AI Weekly, Oct 2026
• Spec-Driven Development for Agentic Software Engineering: Harnessing Human-Agent Teamwork, arXiv, Sep 2026
• How to Write Acceptance Criteria an AI Agent Can Verify, BrainGrid, 2026

SWEBOK v4, Software Requirements, Section 4.3: Acceptance Criteria-Based Requirements Specification: covers how requirements expressed as testable conditions connect to validation, and how acceptance criteria are written and checked.

OWASP Top 10 for LLM Applications 2025, LLM06: Excessive Agency: covers what happens when an AI system has more scope than the task requires and no precise boundaries to check itself against; vague criteria leave the agent's permitted scope undefined.