Debugging and Evaluating Agent Behavior

Fundamentals' Chapter 3 established "trust but verify" — an agent's report describes intent, not a guarantee. This chapter covers what happens once verification actually turns up something wrong: isolating where in a chain the mistake originated, telling a bad brief apart from a genuinely unreliable agent, and evaluating behavior systematically rather than reacting to one incident.

Isolating Where in a Chain Something Went Wrong

For a single agent, "trust but verify" means checking that one agent's own work. In a chain, a wrong final result could have originated at any step — the research agent found the wrong thing, the planning agent proposed a flawed approach despite correct research, or the coding agent implemented a good plan incorrectly. Debugging a chain means checking each step's own actual output against what it should have produced, rather than assuming the last agent in the chain is automatically the one at fault simply because its output is what surfaced the problem.

Distinguishing a Bad Input From a Bad Agent

An agent — even a genuinely well-designed one — given a poor brief, or working from an earlier step's own already-wrong output, will produce a wrong result. That's not the agent's own fault; it's a symptom of a problem that originated earlier in the chain and simply propagated forward, exactly as Chapter 7 described. Before concluding an agent itself is unreliable, confirm what it was actually working from was correct in the first place.

Evaluating Agent Behavior Systematically

Beyond debugging one failure, genuinely evaluating whether an agent — or a whole pipeline — is reliable over time benefits from a small set of representative test cases with known-correct expected outcomes, run repeatedly, rather than judging reliability from a handful of anecdotal results noticed in passing. This applies Fundamentals' Chapter 7's testing discipline to the agents themselves, not just to code.

When to Revise a Brief vs. When to Revise the Agent's Own Definition

If the same agent definition, given a new and better brief, now produces a correct result, the original problem was the brief — a per-task fix, not a sign anything is wrong with the agent itself. If the same agent, given an already-good brief, still produces the wrong kind of result repeatedly, the problem more likely lives in the agent's own definition — its system prompt, tool access, or model choice — and needs revising at that level, rather than re-briefed differently every single time it's used.

Hands-On Exercises