In How to use Claude Code to QA your website I walked through the setup: Claude Code drives agent-browser through the dialer, watches the calls resolve, and files a GitHub issue for anything that doesn't match the expected behavior written into CLAUDE.md.

That post ends where the interesting part starts. A one-shot pass is a demo. Put it on a cron job and you find out what you actually built.

Here's the failure you hit first: the pass runs every morning, the same selector is still broken, and you get a fresh issue for it every single day. Same title, same screenshot, different number. Within a week the qa-bot label is a landfill and you've stopped reading it.

A QA agent that files issues is easy. A QA loop is the machinery around it that makes the queue drain instead of grow.

Four pieces

The loop has four parts, and only one of them is the agent:

1. A spec the agent grades against, in version control.
2. A run that produces evidence for every check, pass or fail.
3. A reconciler that compares this run's failures to the open issues.
4. A close step that shuts issues when the flow goes green again.

The agent does step 2. Steps 1, 3 and 4 are boring scripts, and that's the point. You don't want a language model deciding whether two failures are the same failure.

Write the spec as a file

Put one file per flow in qa/flows/. Each check gets a stable id, because that id is what the whole loop hangs off.

<!-- qa/flows/dialer-test-list.md -->
# Flow: dialer-test-list

Log in with the `staging` auth profile. Go to Dialer, select the "Test"
list, hit Dial. Only ever dial the Test list.

## Checks

- id: voicemail-detect
  expect: Call 1 (voicemail number) ends in state `voicemail`, call is
  logged, dialer advances to the next lead.

- id: ivr-detect
  expect: Call 2 (IVR number) ends in state `ivr-detected`.

- id: gatekeeper-route
  expect: Call 3 (gatekeeper number) loads the gatekeeper talk track.

- id: console-clean
  expect: Zero console errors and zero page errors across the run.

Two things fall out of writing it this way. The expected behavior lives next to the code it describes, so when IVR handling changes the check changes in the same PR. And ivr-detect is now a name that survives across runs, which is what lets you tell "still broken" from "broken again."

Have the agent return JSON

The agent's job is to drive the browser and report what happened. It doesn't file anything. Run it headless and make it hand back a structured result.

Write the contract as a type first, so the prompt and the reconciler can't drift apart:

// qa/types.ts
export type CheckStatus = "pass" | "fail";

export interface CheckResult {
  id: string;
  status: CheckStatus;
  expected: string;
  actual: string;
  console?: string;
  screenshot?: string;
}

export interface RunResult {
  flow: string;
  run: string;
  checks: CheckResult[];
}

Then the runner:

#!/usr/bin/env bash
# qa/run.sh
set -euo pipefail

FLOW="${1:?usage: run.sh <flow-name>}"
RUN_ID="$(date +%Y%m%d-%H%M%S)"
OUT="qa/runs/$RUN_ID"
mkdir -p "$OUT"

claude -p "$(cat <<EOF
Run the QA flow described in qa/flows/$FLOW.md using agent-browser.

For every check in that file, record whether it passed. On failure,
screenshot to $OUT/<check-id>.png and capture console output.

Write your result to $OUT/result.json and nothing else. It must match
the RunResult type in qa/types.ts:
{"flow":"$FLOW","run":"$RUN_ID","checks":[
  {"id":"...","status":"pass"|"fail",
   "expected":"...","actual":"...",
   "console":"...","screenshot":"$OUT/<id>.png"}]}

Do not file issues. Do not fix anything. Report only.
EOF
)" --allowedTools "Bash,Read,Write" > "$OUT/agent.log" 2>&1

bun qa/reconcile.ts "$OUT/result.json"

The Do not file issues. Do not fix anything. line matters more than it looks. Give an agent a gh binary and a bug and it will start improvising a workflow. Keep the reporting job and the filing job in different processes.

There's a safety version of this too. The dialer's QA agent only ever touches the Test list, because these are real calls going out over Twilio. An agent told to both find problems and fix them has a reason to try one more thing, and one more thing is the last behavior you want near a phone system. Splitting the jobs means the process driving the browser has no write access to anything except a JSON file.

Fingerprints are the whole trick

Deduping issues by title fails immediately, because the agent writes a slightly different title every run. Dedupe on an id you control, and hide it in the issue body where GitHub search can still find it:

// qa/fingerprint.ts
import { createHash } from "node:crypto";

export const fingerprint = (flow: string, checkId: string): string =>
  createHash("sha1").update(`${flow}:${checkId}`).digest("hex").slice(0, 12);

export const marker = (fp: string): string => `<!-- qa-fp:${fp} -->`;

Flow plus check id. Nothing from the run itself: not the timestamp, not the error message, not the build sha. If the same check fails for a slightly different reason tomorrow, that's the same broken thing, and it belongs in the same thread.

Reconcile before you file

Now the part that keeps the queue honest. For each check, there are four cases, and three of them are not "open a new issue":

// qa/reconcile.ts
import { $ } from "bun";
import { fingerprint, marker } from "./fingerprint";
import type { CheckResult, RunResult } from "./types";

const result: RunResult = await Bun.file(process.argv[2]).json();

const open: { number: number; body: string }[] =
  await $`gh issue list --label qa-bot --state open --limit 200 --json number,body`.json();

const openByFingerprint = new Map<string, number>();
for (const issue of open) {
  const match = issue.body.match(/<!-- qa-fp:([a-f0-9]{12}) -->/);
  if (match) openByFingerprint.set(match[1], issue.number);
}

for (const check of result.checks) {
  const fp = fingerprint(result.flow, check.id);
  const existing = openByFingerprint.get(fp);

  if (check.status === "fail" && existing === undefined) {
    const title = `QA: ${result.flow} / ${check.id}`;
    const url = await $`gh issue create --title ${title} --label qa-bot --body ${issueBody(result, check, fp)}`.text();
    await $`gh issue edit ${url.trim()} --add-assignee @me`.quiet();
  } else if (check.status === "fail") {
    await $`gh issue comment ${existing} --body ${`Still failing on run ${result.run}.\n\n${evidence(check)}`}`.quiet();
  } else if (existing !== undefined) {
    await $`gh issue close ${existing} --comment ${`Passing again as of run ${result.run}. Closing.`}`.quiet();
  }
}

function evidence(check: CheckResult): string {
  return [
    `**Expected:** ${check.expected}`,
    `**Actual:** ${check.actual}`,
    check.console ? `**Console:** ${check.console}` : null,
    check.screenshot ? `**Screenshot:** ${check.screenshot}` : null,
  ]
    .filter(Boolean)
    .join("\n");
}

function issueBody(run: RunResult, check: CheckResult, fp: string): string {
  return `${evidence(check)}\n**Repro:** ${run.flow}, check ${check.id}, run ${run.run}\n\n${marker(fp)}`;
}

Two things Bun is doing for you here. It runs the TypeScript directly, so there's no build step between a cron firing at 6am and the reconciler executing, and no tsc output to keep in sync. And $ escapes every interpolated value, which matters more than it sounds: issueBody is built partly from text an agent wrote, and it goes straight onto a command line. A console error containing a backtick shouldn't be able to run anything.

The auto-close is the piece people leave out, and it's what turns the label from an archive into a worklist. If nothing in qa-bot is open, the flows are green. That's a status board you can trust because nothing writes to it by hand.

The evidence has to go somewhere

gh has no supported way to attach a binary to an issue, so the screenshot needs a home. Two options that actually work:

For a public repo, push the run directory to an orphan branch and link the raw URL:

git switch --orphan qa-artifacts 2>/dev/null || git switch qa-artifacts
cp -r "$OUT" "artifacts/$RUN_ID"
git add artifacts && git commit -qm "qa run $RUN_ID" && git push -q origin qa-artifacts
git switch -

For a private repo, raw URLs need auth and the images won't render, so put them in a bucket instead. I use R2 with a short retention rule, because QA screenshots from three weeks ago have never once been useful.

Either way, the body the reconciler writes keeps the shape from the original post, plus the marker:

**Expected:** Dialer detects the IVR menu and flags the call as IVR.
**Actual:** Call sat in "connected" for 45s, logged as a completed conversation.
**Console:** TypeError: Cannot read properties of undefined (reading 'digits') - ivr-detect.ts:84
**Screenshot:** https://qa.example.com/20260830-0600/ivr-detect.png
**Repro:** Test list, call 2 of 3, staging build 2f41c9a

<!-- qa-fp:8c41d2a09b17 -->

Two guards you'll want by week two

Require two consecutive failures before filing. Browser flows are flaky, and a loop that files on the first red will file noise. Keep the last run's result.json, and only open an issue when the same fingerprint failed twice in a row. Comment on the second failure of an already-open issue, always.

Cap the blast radius. Add a hard limit on new issues per run. If a deploy breaks the login page, every downstream check fails, and you do not want 30 tickets that all say "couldn't log in." When the cap trips, file one issue saying the run collapsed and stop.

const failures = result.checks.filter((c) => c.status === "fail");
if (failures.length > 5) {
  await fileRunCollapse(result, failures);
  process.exit(1);
}

Five is arbitrary. Pick the number from how many checks the flow has, and keep it well under the total. A run where half the checks fail is almost always one upstream cause wearing several costumes, and the reconciler has no way to work out which one.

Where the loop still needs you

The reconciler can tell you a check went from red to green. It has no idea whether the check was right. If ivr-detect asserts the wrong end state, the loop will confidently keep it green forever, and you'll find out from a customer.

So the spec files are the part I still edit by hand, one corrected assumption at a time. Everything downstream of them is plumbing, and plumbing is exactly what I want an agent inside of: it drives the browser, it writes down what it saw, and a script with no opinions decides what that means.

If you want the longer argument for this shape, QA in the Era of AI is the field guide version, and Context Engineering for Production Agents covers why the verification has to live outside the agent's own summary of its work.