"Done! I've implemented the refund approval flow and all tests are passing."

No test ran. The agent didn't lie, exactly. It predicted the sentence that usually follows work like that, and the sentence is free. Saying "done" costs a model nothing, which means it carries no information.

In Context Engineering for Production Agents I said a production agent needs an externally observable definition of done. That was one bullet in a long post. It deserves the whole thing, because it's the single most useful artifact you can add to an agentic workflow, and hardly anyone writes one down.

A definition of done has four parts. All four have to be machine-checkable, or you've written a wish.

1. Source boundaries

What the agent is allowed to treat as truth. Without this, an agent that can't find the answer will invent a plausible one from training data, and a plausible answer about your codebase is worse than no answer.

sources:
  authoritative:
    - src/db/schema.ts       # schema truth, Directus owns the DB
    - docs/refund-policy.md  # business rules
  reference:
    - src/services/**        # patterns to follow, not rules to cite
  forbidden:
    - node_modules/**
    - "training data recall for any API in this repo"
  on_missing: stop_and_ask

on_missing: stop_and_ask is the important line. The default agent behavior when a fact is missing is to guess, and guessing is a failure mode you have to explicitly turn off.

2. Acceptance criteria, written as observations

The rule: every criterion describes something you could see from outside the process. No criterion may describe the agent's confidence, effort, or intent.

Bad, because none of it is observable:

- Refund approvals are properly implemented
- Code follows existing patterns
- Error handling is comprehensive

Good:

acceptance:
  - id: reject-untokened
    observe: "POST /api/refunds/execute without approvalToken returns 403"
  - id: writes-audit-row
    observe: "A successful execute inserts one row into refund_approvals
              with approver_id, payload_hash, created_at"
  - id: no-schema-migration
    observe: "git diff --name-only contains no files under migrations/"

Each one has an id, and each id maps to a command in the next section. If you can't write the command, the criterion is still a wish.

3. Verification commands the agent doesn't get to run

Here's the part that changes behavior. Verification gets reported by a runner, and the agent has to carry the runner's output into its final answer verbatim.

verify:
  - id: build
    run: bun run build
    expect: exit 0
  - id: reject-untokened
    run: bun test refunds/execute-without-token
    expect: exit 0
  - id: writes-audit-row
    run: bun run scripts/check-audit-row.ts
    expect: stdout contains "AUDIT_ROW_OK"
  - id: no-fake-done
    run: bin/no-fake-done.sh
    expect: exit 0

The runner is boring on purpose:

// bin/verify.mjs
import { readFileSync } from 'node:fs'
import { execSync } from 'node:child_process'
import { parse } from 'yaml'

const dod = parse(readFileSync(process.argv[2], 'utf8'))
const results = []

for (const check of dod.verify) {
  let out = '', code = 0
  try {
    out = execSync(check.run, { encoding: 'utf8', stdio: 'pipe' })
  } catch (e) {
    code = e.status ?? 1
    out = `${e.stdout ?? ''}${e.stderr ?? ''}`
  }
  const pass = check.expect.startsWith('stdout contains')
    ? out.includes(check.expect.split('"')[1])
    : code === 0
  results.push({ id: check.id, pass, code, tail: out.trim().split('\n').slice(-8) })
}

const verdict = results.every(r => r.pass) ? 'DONE' : 'NOT_DONE'
console.log(JSON.stringify({ verdict, results }, null, 2))
process.exit(verdict === 'DONE' ? 0 : 1)

And the instruction that ties it together, in CLAUDE.md:

## Definition of done

Work is done when `node bin/verify.mjs .work/dod/<issue>.yaml` prints
`"verdict": "DONE"`.

You must run it and paste its full JSON output as the last block of your
final message. If you did not run it, say so and stop. Never describe a
check as passing without that output above your claim.

The trick is that the agent can no longer produce a completion claim by itself. It has to fetch one. That's a much harder thing to hallucinate than an adjective.

One placement note, because it decides whether this sticks. Put the instruction in CLAUDE.md, not in the task prompt. An agent carrying the rule in standing context will run the verifier on its own at the end of a task. An agent told once at the top of a long session has compacted that instruction away by the time it matters, and you're back to reminding it, which is the job you were trying to stop doing.

4. What the agent must refuse to claim

The last section is a short list of sentences the agent is not allowed to write without specific evidence attached. Keep it under six lines, because a list of thirty NEVERs gets ignored wholesale.

refuse_to_claim:
  - claim: "tests pass"
    requires: pasted output of the test command, including the counts
  - claim: "the bug is fixed"
    requires: the reproduction, run again, now failing to reproduce
  - claim: "deployed"
    requires: a URL and its status code
  - claim: "no other code is affected"
    requires: output of a grep for the changed symbol

Every one of these is a sentence I've read from an agent that turned out to be false. They're all cheap to prove and expensive to skip.

The fake-done grep

The other thing worth wiring in, because it's the most common way work arrives looking finished when it isn't:

#!/usr/bin/env bash
# bin/no-fake-done.sh
set -uo pipefail
FILES=$(git diff --name-only origin/main...HEAD | grep -E '\.(ts|tsx|js|astro)$' || true)
[ -z "$FILES" ] && exit 0

HITS=$(grep -nE 'test\.(skip|only)|it\.(skip|only)|describe\.(skip|only)|TODO: implement|throw new Error\("[Nn]ot implemented' $FILES || true)
if [ -n "$HITS" ]; then
  echo "FAKE DONE:"; echo "$HITS"; exit 1
fi
exit 0

A skipped test, a stubbed branch, and a TODO all read as finished code if you're skimming a 400 line diff at the end of a day. Make the build fail on them instead.

Why this is a file

You could put all of this in a system prompt. It would work for a while and then quietly stop, because a prompt is advice and a script is a wall.

The other reason is versioning. The definition of done lives in the repo next to the code it describes, so when the refund rules change, the observe lines change in the same PR. Six months later you can read what "done" meant when that feature shipped, which is the same reason you keep tests around.

One per issue, written before the work starts. Writing it is where you find out the ticket is underspecified, which is worth the ten minutes on its own.

The honest limit

A definition of done catches an agent claiming work it didn't do. It cannot catch a criterion that's wrong. If reject-untokened asserts a 403 and the correct behavior is a 401, the verifier will go green forever and you'll find out from an integration partner.

So the criteria are the part I write by hand, and the part I actually review. Everything downstream is machinery. That's the deal I've settled on with agents generally: they do the work, a script with no opinions decides whether the work happened, and I own what "happened" means.

Eval-Driven Development goes deeper on the grading side, and QA in the Era of AI covers checking the running app rather than the diff.