How to Build an Agent Eval Dataset from Real Tasks, Trajectories, and Regression Tests

Last week, the daily-report agent generated reports normally. After a model upgrade, it still returned a final summary that looked correct, but the trajectory skipped the permission check and data refresh tools. A support agent gave the right final reply, yet called the refund tool with the wrong arguments; a public benchmark would not catch that. An incident review already has a traceId, tool call, approvalId, state event, and human judgment, but none of it has been turned into a test that must run before the next release.

You cannot cover this kind of regression by swapping models or running one more demo. An agent eval dataset should come from the real task distribution and production failure trajectories: tool calls, argument shapes, state transitions, approval records, and human labels. Then it has to run in CI as a regression gate. The focus here is concrete failure cases and real scenarios: how to build a durable eval dataset that protects the quality boundary of every release.

1. Why public benchmarks are not enough

1.1 The limits of public benchmarks

Public benchmarks measure model capability, not the quality of your business tasks. Evaluation sets such as AgentBench and WebArena provide standardized tasks, but they cannot cover your team’s toolchain, permission model, and state machine.

For example, your reporting agent may need to call check_permission, refresh_data, and generate_report in the order “check → refresh → generate.” A public benchmark will not test that flow because it does not know your tool schemas or permission boundaries.

A good public benchmark score does not mean your business task will not regress. An agent eval dataset should test your real task distribution, not general reasoning ability.

1.2 The core value of a private eval dataset

A private eval dataset catches risks that public benchmarks cannot see. The table below shows typical cases:

Use public benchmarks to choose models; use your own dataset to protect release gates. They complement each other, but one cannot replace the other. If you have already read Agent Evaluation Benchmarks, you know which public evaluation sets help with model selection. The focus here is how to build evaluation data from your own production environment.

2. Core fields in an agent eval dataset

2.1 Required fields

An agent eval dataset is not just a list of user inputs and expected answers. It needs to record tool-call trajectories, state transitions, and human judgments. These are the required fields:

LangSmith’s Evaluation concepts recommends covering tool selection, argument formatting, and execution trajectory for agent evaluation instead of judging only the final answer. OpenAI Agents SDK Tracing provides fields such as trace_id, workflow_name, and span to record tool calls and state events, but those raw fields need redaction and extraction before they enter a dataset.

2.2 Optional fields

Add the following fields when your business needs them:

Alibaba Cloud - Top choice for developers, exclusive 15% off for Lighthouse servers

Get 15% Discount →Ad

If you have read the agent permission model article (N157) and the human approval design article (N155), you know that approval records should capture the approval step, approver, and result so high-risk actions can be verified as approved.

2.3 Redaction and privacy

Production traces cannot enter the evaluation set as-is. The OpenAI Agents SDK Tracing docs note that some spans can capture model inputs and outputs or function tool inputs and outputs, so you need to configure whether sensitive data is included.

Fields that must be redacted: PII (names, emails, phone numbers), secrets (API keys, database passwords), and customer data (order IDs, transaction amounts).

Fields to keep: traceId, span_id for debugging, workflow_name for categorization, tool names, and state names for assertions.

Redaction methods include replacing values with placeholders such as <user_id>, hashing them, or deleting them outright. Phoenix’s Running Evals on Traces docs show that traces can be turned into evaluation inputs, but they still need labels, scores, and rubrics. Do not put complete raw traces directly into the eval dataset.

3. From production trace to eval example

3.1 Data collection flow

To turn a production trace into an eval example, you need four steps: redaction, labeling, layering, and storage. This is ongoing operations work, not a one-off engineering task.

Checklist:

1. Export failed traces from your monitoring platform. See Agent Monitoring and Failure Recovery.
2. Classify failures by type: tool-call error, argument-format error, state-transition error, or final-answer error.
3. Redact data: remove PII, secrets, and customer data; keep traceId, span_id, and workflow_name.
4. Add human labels: correct, wrong, or partially correct + failure reason.
5. Store by layer: golden set / negative cases / edge cases.

Phoenix datasets documentation notes that examples can be collected from production, staging, evaluations, or manual sources. Production gives you the real task distribution, while staging test cases can fill in boundary scenarios.

3.2 Layering standards

Labeling dimensions, aligned with an LLM-as-judge rubric, include tool selection correctness, argument format correctness, execution order correctness, final answer quality, and alignment with human labels.

OpenAI Evaluation best practices recommend test data that includes typical, edge, and adversarial cases, with human expert labellers. The layering above maps to that guidance, but it emphasizes high-frequency tasks and typical failure modes because the goal is to protect release gates, not cover every possible scenario.

4. How to write trajectory assertions

4.1 Types of trajectory constraints

Trajectory assertions define the tool-call, argument-format, state-transition, and approval rules an agent must follow. These are five common constraint types:

LangSmith’s Evaluation concepts mention trajectory, tool selection, and argument formatting as parts of agent evaluation. Phoenix’s Running Evals on Traces docs use span_id to attach evaluation results to a specific execution segment, which shows how trajectory assertions can point to the failing span.

If you have read the agent planning evaluation article (N158) and the state-machine boundary article, you know that state trajectory fields come from state, event, and checkpoint records. They can verify whether the state transition follows the expected order.

4.2 Code example

Here is pseudocode for trajectory assertions in a general agent evaluation setup:

# trajectory assertions example
def assert_trajectory_correct(result):
    # Extract the tool-call list
    tools_called = [call.tool for call in result.tool_calls]

    # Required tools
    required_tools = ["check_permission", "refresh_data"]
    for tool in required_tools:
        assert tool in tools_called, f"Missing required tool: {tool}"

    # Forbidden tools
    forbidden_tools = ["delete_user"]
    for tool in forbidden_tools:
        assert tool not in tools_called, f"Called forbidden tool: {tool}"

    # Argument format constraints
    for call in result.tool_calls:
        if call.tool == "refund":
            assert isinstance(call.args["refund_amount"], (int, float)), "refund_amount must be number"
            assert len(call.args["user_id"]) == 36, "user_id must be uuid format"

    # State transition order
    expected_states = ["init", "check", "approve", "execute"]
    actual_states = [event.state for event in result.state_events]
    assert actual_states == expected_states, "State transition order mismatch"

    # Approval step existence
    approval_events = [e for e in result.state_events if e.type == "approval"]
    assert len(approval_events) >= 1, "Missing approval step"

This example shows how to check required tools, forbidden tools, argument formats, state transition order, and approval steps. In a real implementation, you can use LangSmith trajectory assertions or Phoenix trace evals, but the core logic is the same.

Railway - Modern deployment platform, zero-config, go live in minutes

Start Now →Ad

If you have read Agent Planning Evaluation, you know that planning trajectories can also become regression samples for task decomposition and self-correction.

5. Layered rubrics for LLM-as-judge

5.1 Why a total score is not enough

When LLM-as-judge returns only a total score, the result is noisy and hard to debug. A score of 80 might mean the tool selection was wrong but the answer happened to be right, or the answer was wrong but the format looked good. You cannot tell whether the issue sits in the prompt, tool schema, retrieval, or state machine.

LangSmith’s Evaluation concepts recommend defining quality standards for key agent components: correct tool selection, proper argument formatting, and trajectory. Scoring each component separately makes it much faster to find regressions after a model upgrade or prompt change.

If you have read Advanced Prompt Engineering Practices, you know that DeepEval also emphasizes layered metrics such as TaskCompletenessMetric instead of judging only the final output.

5.2 Layered scoring table

The rubric below splits agent evaluation into five dimensions, each scored from 0 to 2:

Example rubric configuration for LangSmith or Phoenix:

rubric:
  dimensions:
    - name: tool_selection
      weight: 0.3
      criteria:
        0: "Called the wrong tool"
        1: "Called some correct tools"
        2: "Tool selection is fully correct"

    - name: argument_formatting
      weight: 0.2
      criteria:
        0: "Argument type is wrong"
        1: "Some fields have format errors"
        2: "Argument format is fully correct"

    - name: execution_order
      weight: 0.2
      criteria:
        0: "State transition jumps or repeats"
        1: "Execution order has deviations"
        2: "Execution order is fully correct"

    - name: final_answer_quality
      weight: 0.2
      criteria:
        0: "Does not satisfy the user request"
        1: "Partially satisfies the request"
        2: "Fully satisfies the request"

    - name: human_alignment
      weight: 0.1
      criteria:
        0: "Conflicts with expert judgment"
        1: "Partially aligned"
        2: "Highly aligned"

Tool selection has the highest weight (0.3) because choosing the right tool is central to an agent. Human-label alignment has the lowest weight (0.1) because it is mainly used to calibrate whether the LLM-as-judge agrees with expert judgment.

6. Release gates and regression testing

6.1 CI integration steps

The value of an agent eval dataset comes from running it in CI as a release gate. Use this integration flow:

Checklist:

1. Add an eval run step to your CI pipeline, such as pytest, LangSmith offline eval, or Phoenix experiments.
2. Configure a pass-rate threshold, such as 90%.
3. Require the golden set to pass completely, or 100%.
4. Configure alert rules for negative cases. They may notify the team without blocking.
5. Set up version comparison, such as improvement or regression, against a baseline version and the current version.
6. Define when the gate runs: PR / merge / canary.

LangSmith’s Evaluation concepts mention that offline evaluations can be used for regression testing, unit testing, and backtesting over examples in datasets. Phoenix experiments documentation describes comparing prompt, LLM, or application logic changes to judge improvement or regression. DeepEval describes pytest-native evals that can run in CI/CD or Python scripts, using LLMTestCase and TaskCompletenessMetric to score agent traces layer by layer.

6.2 Threshold configuration table

CI configuration example:

evaluation:
  triggers:
    - pull_request
    - merge_to_main
    - canary_deployment

  conditions:
    pass_rate: 0.9
    golden_set: 1.0
    no_regression: true

  notifications:
    on_failure: ["team-lead", "on-call"]
    on_negative_alert: ["team-lead"]

Gate triggers include PRs, which stop bad code from entering the main branch; merge checks, which ensure the merged state still satisfies the conditions; and canary checks, which validate the change before production traffic. If you have read the cost-control and model-routing article (N156), you know that cost thresholds can also become gate conditions. This piece stays focused on evaluation thresholds.

7. Real case walkthrough

7.1 Scenario

The daily-report agent worked last week: it called check_permission to verify access, called refresh_data to refresh data, and then called generate_report to generate the report. After a model upgrade, the agent still returned a final summary that looked correct, but the trajectory skipped check_permission and refresh_data.

Vultr - High-performance NVMe VPS with 32 global locations, one-click Docker deploy

View Pricing →Ad

The user received a report without permission checks and with stale data, but that was not visible on the surface. A public benchmark would not detect this risk because it does not have your tool schema or state machine.

7.2 Diagnosis

Export the failed trace from the monitoring platform. See Agent Monitoring and Failure Recovery. Trajectory analysis shows that the tool-call list is missing check_permission and refresh_data, the state transition jumps from init directly to execute, the check and approve steps are missing, and the final answer looks right even though the data source was not refreshed.

Label it as: tool selection error + state transition jump.

7.3 Dataset update

Add the failure case to negative cases. Also expand the golden set: find last week’s successful daily-report task, include the complete tool-call sequence, and label it as a correct case.

Update the trajectory assertions: add check_permission and refresh_data to the required tool list, and set the state transition assertion to ["init", "check", "approve", "execute"].

7.4 CI gate interception

The next PR triggers an eval run. The negative case raises an alert: check_permission was not called. The golden set passes completely, so the successful baseline is still intact.

The issue points to the prompt. After the model upgrade, the model interprets “generate a report” as direct execution and ignores the required pre-check and refresh steps.

7.5 Fix and regression

Fix the prompt by stating that the agent must check permissions and refresh data before generating the report. Run the eval again: pass rate improves from 85% to 95%, the negative case passes without an alert, the golden set passes completely, and version comparison shows improvement with no regression.

Merge and release. The CI gate caught this regression before it reached production.

7.6 Field mapping table

This loop shows the full path from production failure to regression test. After each incident review, turn the traceId, tool call, approvalId, state event, and human judgment into an eval example so the next release does not repeat the same failure.

8. What to read next

• Agent state machines and prompt boundaries: where state-transition constraints and trajectory assertion fields come from
• Agent permission models: how approval records and high-risk tool-call rules work
• Human approval design: how to turn approval records and approval flows into durable tests
• Cost control and model routing: how cost thresholds can become gate conditions
• Agent Monitoring and Failure Recovery: how to export failed traces and build an incident review loop