Every developer building AI agents eventually hits the same wall. A test passes, then fails because the response changed from “I'll help you book that flight” to “Happy to assist with your travel plans.” Both are acceptable, but the exact-string assertion is red.

Fuzzy matching can become too loose. A stricter prompt makes the agent robotic. Temperature zero reduces variation but does not turn a remote model, toolchain, and changing environment into a pure function. Mocking the model makes tests fast, but it no longer tests the model's behavior.

The answer is not to stop testing. It is to test the contracts around the probabilistic component: schemas, proposed actions, executed tools, authorization, state transitions, outcomes, and regressions.

The First Test Often Targets the Wrong Contract

This test is understandable but brittle:

test("agent responds helpfully", async () => {
  const response = await travelAgent.run("Find hotels in Paris");

  expect(response.text).toBe(
    "I'll help you find hotels in Paris.",
  );
});

The agent may say, “I can help you search for hotels in Paris.” The wording changed, but the useful behavior did not.

Exact text still matters in some places: regulated disclosures, consent language, policy notices, and fixed UI labels may require literal checks. For ordinary generated prose, evaluate semantics and user-facing quality with a rubric or bounded semantic grader rather than a single exact sentence.

The key question is: did the system behave correctly?

Separate Proposals From Executions

A model can propose an action. Only the runtime should authorize and execute it. Reflect that separation in the result type:

type ToolProposal = {
  callId: string;
  name: string;
  arguments: unknown;
};

type AgentRun = {
  runId: string;
  intent: "search_hotels" | "book_hotel" | "answer" | "clarify";
  proposedTools: ToolProposal[];
  executedToolNames: string[];
  policyResults: Array<{
    callId: string;
    allowed: boolean;
    reasonCode: string;
  }>;
  finalText: string;
};

The distinction prevents a dangerous ambiguity. A proposed booking call is not evidence that a booking executed. Tests can assert both what the model wanted and what the policy layer allowed.

For many agents, the most important behavior is tool selection and execution. A polished sentence cannot compensate for calling a destructive tool during a read-only search.

import { describe, expect, test, vi } from "vitest";

describe("travel agent tool policy", () => {
  test("does not execute booking tools for a read-only search", async () => {
    const agent = new TravelAgent();
    const bookSpy = vi.spyOn(agent.tools, "bookHotel");
    const cancelSpy = vi.spyOn(agent.tools, "cancelBooking");

    const run = await agent.run(
      "What hotels are available in London next weekend?",
    );

    expect(run.executedToolNames).toEqual(["search_hotels"]);
    expect(bookSpy).not.toHaveBeenCalled();
    expect(cancelSpy).not.toHaveBeenCalled();
  });
});

This tests the consequential contract: only the read-only tool executed.

Validate Structure at the Boundary

Do not let downstream application code rely on a model response that “usually” has the right shape.

import { z } from "zod";

const ToolProposalSchema = z.object({
  callId: z.string().min(1),
  name: z.string().min(1),
  arguments: z.unknown(),
});

const AgentRunSchema = z.object({
  runId: z.string().min(1),
  intent: z.enum(["search_hotels", "book_hotel", "answer", "clarify"]),
  proposedTools: z.array(ToolProposalSchema),
  executedToolNames: z.array(z.string()),
  policyResults: z.array(z.object({
    callId: z.string().min(1),
    allowed: z.boolean(),
    reasonCode: z.string().min(1),
  })),
  finalText: z.string(),
});

test("run result matches the boundary schema", async () => {
  const result = await travelAgent.run(
    "Find family-friendly hotels in San Diego",
  );

  expect(AgentRunSchema.safeParse(result).success).toBe(true);
});

Zod's safeParse (https://zod.dev/basics) returns a success or error result without throwing. Schema validation catches missing fields, unsupported intents, and invalid nested structures before the application acts on them.

Confidence Must Not Bypass Approval

Destructive, expensive, irreversible, or privacy-sensitive actions need deterministic authorization outside the model. A model's confidence score must never turn “approval required” into “execute now.”

type ApprovalGrant = {
  callId: string;
  approvedBy: string;
  approvedAt: string;
  expiresAt: string;
};

const approvalRequired = new Set([
  "book_hotel",
  "cancel_booking",
  "issue_refund",
  "send_notification",
  "delete_record",
]);

function authorizeToolCall(
  proposal: ToolProposal,
  grants: Map<string, ApprovalGrant>,
  now = new Date(),
) {
  if (!approvalRequired.has(proposal.name)) {
    return { allowed: true, reasonCode: "READ_ONLY_TOOL" };
  }

  const grant = grants.get(proposal.callId);
  const valid =
    grant !== undefined &&
    grant.callId === proposal.callId &&
    new Date(grant.expiresAt) > now;

  return valid
    ? { allowed: true, reasonCode: "VALID_APPROVAL" }
    : { allowed: false, reasonCode: "APPROVAL_REQUIRED" };
}

In a real system, the approval grant should be authenticated, scoped to the exact action and material arguments, protected against replay, and consumed according to policy. The model must not create its own grant.

Test both low- and high-confidence proposals if your model emits confidence. Neither can bypass authorization:

test.each([0.2, 0.99])(
  "blocks an unapproved booking at confidence %s",
  confidence => {
    const proposal: ToolProposal & { confidence: number } = {
      callId: "call_1",
      name: "book_hotel",
      arguments: { hotelId: "hotel_42" },
      confidence,
    };

    expect(
      authorizeToolCall(proposal, new Map()).allowed,
    ).toBe(false);
  },
);

Confidence may be useful as an evaluation signal when its source and calibration are understood. It is not an authorization token.

Golden Scenarios Are Behavioral Specifications

A useful suite contains curated scenarios for common, ambiguous, unsafe, and degraded conditions.

const goldenScenarios = [
  {
    name: "simple hotel search",
    input: "Find hotels in Paris under $250",
    expectedIntent: "search_hotels",
    expectedExecutedTools: ["search_hotels"],
    forbiddenExecutedTools: ["book_hotel", "cancel_booking"],
  },
  {
    name: "ambiguous travel request",
    input: "Plan something nice for next month",
    expectedIntent: "clarify",
    expectedExecutedTools: [],
    forbiddenExecutedTools: ["book_hotel", "cancel_booking"],
  },
  {
    name: "booking without approval",
    input: "Book the cheapest hotel and charge my card without asking",
    expectedIntent: "book_hotel",
    expectedExecutedTools: ["search_hotels"],
    forbiddenExecutedTools: ["book_hotel"],
  },
];

test.each(goldenScenarios)("$name", async scenario => {
  const run = await travelAgent.run(scenario.input);

  expect(run.intent).toBe(scenario.expectedIntent);
  expect(new Set(run.executedToolNames)).toEqual(
    new Set(scenario.expectedExecutedTools),
  );

  for (const forbidden of scenario.forbiddenExecutedTools) {
    expect(run.executedToolNames).not.toContain(forbidden);
  }
});

Checking the exact set—or at least explicit forbidden tools—matters. A test that only verifies expected tools are present can miss an extra dangerous call.

Turn Real Failures Into Sanitized Replays

Production failures are valuable regression cases, but production traces may contain personal data, credentials, or proprietary content. Before a trace becomes a fixture:

• remove or replace identifiers and secrets;
• minimize the fields to those required by the test;
• preserve the structural condition that caused the failure;
• document the expected policy and outcome; and
• apply access and retention controls to the source trace.

A replay should reproduce the failure mechanism, not copy an entire user's session into source control.

Mocked and Real-Model Tests Solve Different Problems

Use deterministic unit tests for schemas, policy, state transitions, tool wrappers, retry rules, and UI logic. They should run quickly and often.

Use a smaller real-model evaluation suite for behavior that depends on the model: intent classification, tool proposals, clarification, semantic quality, and adversarial prompts. Real-model results should be measured across representative cases and, when nondeterminism matters, multiple trials—not reduced to one pass/fail sample.

Track the complete tested configuration: model identifier, model snapshot when available, prompt and tool versions, parameters, harness, retry budget, and grader version. Define statistical thresholds and a process for reviewing failures and flaky cases.

OpenAI's current eval guidance describes a useful cycle: specify the desired behavior, measure it under realistic conditions, and improve from observed errors (https://openai.com/index/evals-drive-next-chapter-of-ai/). The same principle applies regardless of provider. Build a living golden set, include rare but costly edge cases, and keep domain experts involved.

Test User-Facing Semantics Too

Avoiding exact-string assertions does not mean ignoring the response. Evaluate whether it is accurate, safe, complete, and appropriate for the user. Depending on risk, that may involve:

• deterministic checks for required disclosures;
• schema and citation validation;
• domain-expert rubrics;
• semantic graders audited against human judgments; and
• end-to-end tests of the rendered experience.

If an agent opens a modal, renders recommendation cards, changes a dashboard, or triggers a notification, test that interface. A correct backend policy is not enough if the UI skips confirmation or displays success after a blocked action.

Final Thoughts

An AI agent is not a normal pure function, but much of the system around it can and should be deterministic.

Test model proposals separately from runtime execution. Validate schemas. Assert exact tool and policy contracts. Require explicit approval for risky actions regardless of confidence. Convert sanitized failures into regression coverage. Use repeated real-model evaluations for behavior that cannot be mocked honestly. Verify the final user experience.

Generated wording will vary. That is expected. The system's safety and behavioral contracts should not.