Verification and evaluation

How to check that the agent did the job right without taking its word for it: give it ways to verify itself, review both the result and the process, and compare setups honestly.

Before you start

  • Task framing for agentsTechnique

    How to turn an idea into a request the agent can carry out and you can review: goal, context, boundaries, expected result, and how to know it is done.

    Published

Key idea: “I’m done” is a claim, not proof. Give the agent ways to check its own work —tests, type checks, a linter— and check the result yourself with evidence, not with its summary.

The problem

Agents produce results that look correct. Sometimes they are. Other times they hide a silent bug, a file that shouldn’t have been touched, or a test that “passes” because the agent changed it so it would pass. If you trust the final summary, you will find the problem late.

Two different ideas

  • Verifying means checking whether this result meets this task. Do the tests pass? Were the boundaries respected?
  • Evaluating means studying how a setup behaves across several cases to decide whether a change to the harness —a new AGENTS.md, a Skill— makes it better or worse.

You need verification in every task. You need evaluation when you want to compare setups.

How verification works

1. Let the agent verify itself

One of the most useful improvements to a harness is giving the agent checks it can run on its own: tests, type checking, a linter, a build. With them, the agent tests its work, sees the error, and fixes it before handing it to you. Without them, it can only assume things work.

That is why your verification commands belong in your AGENTS.md, and the acceptance criteria of each task should be checkable with them.

2. Check the result, not the summary

Before accepting a change:

  • Read the diff. Does it only touch what it should? Did any existing test change?
  • Run the checks yourself, or confirm they were actually run.
  • Go through the task’s acceptance criteria one by one.

3. Review the process too

The final result doesn’t tell the whole story. The session history —which files it read, which commands it ran, which errors it hit— tells you how it got there. When something goes wrong, the clue is usually in that history: did it read an old version of a file? Ignore a tool error? Skip a step? Keep it when you want to learn from a failure.

4. What a test can’t see

Some things can’t be checked automatically: whether the design is right, whether the code will be maintainable, whether the solution solves the real problem. That still needs human review. A second reviewer agent can help, but it also makes mistakes and needs clear criteria.

Practical example

For the summarizeReading task, turn each criterion into a concrete check:

Criterion How to check it
countWords still works Existing tests pass and the diff doesn’t modify them
Counts words with repeated spaces Test with " hello world " → 2
Extracts the first sentence Tests with ., !, ?, and no final punctuation
Respects the boundaries The diff doesn’t touch README.md or package.json
Everything passes node --test finishes without errors

The table doesn’t measure whether the agent is “good” in general. It states what evidence you need to accept this task.

Evaluating setups

If you want to know whether a change to the harness helps, compare honestly:

  • Keep everything the same except one thing: the same model, the same task, and the same project.
  • Save the results exactly as they came out, before fixing them. If you fix something by hand before recording it, you are no longer measuring the setup.
  • Keep failures and doubtful results too. They are part of the evidence.
  • Don’t generalize from a single case. One run shows what happened once, not a rule.

Common mistakes

  • Accepting “all tests pass” without checking. Run the tests or look at the real output.
  • Not checking whether it changed the tests. An agent can “fix” a failing test by modifying the test.
  • Confusing “it works” with “it’s right”. Compiling doesn’t mean it meets the task.
  • Tests that test nothing. A test that always passes gives false confidence. Check that it fails when the code is wrong.
  • Asking the same agent to evaluate itself and simply believing it. Self-review helps catch mistakes, but it doesn’t replace an independent check.

When to use it

Whenever the result matters. Start with the minimum: a test command the agent can run and a review of the diff. Add more checks when you see the same kind of error repeat.

Next step

Asking the agent to verify itself works almost always, but not always. Hooks explains how to make checks run on their own, without relying on the agent to remember.

Sources

Accessed on 2026-10-05.

  • State and memoryTechnique

    How to make a long task survive the end of a session: keep progress outside the model, in files and in Git, and tell task state apart from long-term memory.

    Published
  • Hooks: checks that run on their ownTechnique

    Commands that run automatically at specific moments of the agent’s work, such as before editing a file or when it finishes, so the important steps do not depend on the agent remembering them.

    Published