Reliable agent execution trajectories require deterministic grading, complete failure records, and traceable source lineage. Gold Trace Dataworks builds these datasets by preserving the full history of simulated runs, including both successful recoveries and exhausted retries. This approach ensures that production models learn from verified evidence rather than curated successes alone, reducing the risk of hallucinated behaviors in live environments.
Deterministic Grading Over Model Opinions
Model-based grading introduces variance that can corrupt training data. A deterministic grader is a script that returns the same verdict for the same output every time. In the Tool Failure Recovery v2 study, GTDataWorks used deterministic code to grade 629 recorded attempts. This method ensures that a result can be re-checked by someone who was not in the room. When a grader cannot decide a verdict in code, the attempt goes to human review, and that review is recorded alongside the receipt. This distinction prevents the introduction of subjective bias into the training set.
Including Failures in the Training Set
Excluding failures creates a biased view of agent capability. In the 500-scenario evaluation, 447 scenarios met the recovery criteria, while 53 exhausted their retry budget. The training projection selects one successful example per accepted scenario, but the evaluation retains the full population. This distinction is critical for production reliability. If a model is trained only on successes, it may not learn how to handle edge cases where tools fail. By keeping failed scenarios visible in the denominator, developers can understand the true limits of the agent's recovery protocol.
Traceable Source Lineage and Evidence
Source lineage is the documented history of how a data point was created and verified. GTDataWorks links every normalized record to source hashes and evaluation information. This allows teams to revisit a conclusion without treating a summary as the original event. The record contains 683 total receipt records, including 629 model-attempt records and 54 synthetic terminal markers. Preserving these separately lets a training view change without rewriting the historical record. This transparency is essential for auditing AI behavior in regulated industries.
Respecting Policy and Authority Boundaries
Agents must operate within defined authority boundaries. In the permission-denial template, the primary cohort produced no accepted recovery. A separate nine-attempt diagnostic also produced no accepted recovery across its 50 scenarios. All 450 diagnostic attempts recorded the strategy to respect permission boundaries. This outcome does not establish permission bypass; rather, it confirms that the agent adhered to its constraints. One inspected blocked response failed schema checks for a missing required field. This distinction matters because a rubric failure can expose an output-contract problem rather than a safety violation.

Preparing for Production Deployment
Production readiness requires more than a high success rate. It requires a retained-proof report that a stranger can re-check. This report includes a finding ledger with IDs and dispositions, an evidence manifest with SHA-256 hashes for every file, and a negative control. The negative control is a deliberately wrong submission that the graders must fail. If the graders pass the negative control, the evaluation is invalid. This level of rigor ensures that the model's behavior is not just statistically likely, but verifiably correct. Teams can use this evidence to justify deployment decisions to stakeholders.
Key Takeaways
- Deterministic grading ensures consistent and reproducible evaluation results.
- Including failed scenarios in the evaluation provides a realistic view of agent limits.
- Source lineage allows teams to trace every data point back to its original evidence.
- Respecting policy boundaries is a critical safety feature for production agents.
- A retained-proof report with a negative control validates the evaluation process itself.
- Separating evaluation evidence from training projections prevents data corruption.
Frequently Asked Questions
Why is deterministic grading important for agent evaluation?
Deterministic grading ensures that the same input always produces the same verdict. This eliminates the variance introduced by model-based graders, making the evaluation results reproducible and auditable.
Should failed agent runs be included in training data?
Failed runs should be included in the evaluation set to understand the agent's limits. For training, a projection may select only successful examples, but the full record must be retained for context and auditing.
What is a retained-proof report?
A retained-proof report is a document that includes a finding ledger, an evidence manifest with hashes, and a negative control. It allows a third party to re-check the evaluation results without access to the original system.
How does source lineage improve data reliability?
Source lineage documents the history of how a data point was created. It allows teams to verify that a conclusion is supported by the original evidence, rather than a potentially altered summary.
What is a negative control in agent evaluation?
A negative control is a deliberately wrong submission that the graders must fail. If the graders pass the negative control, it indicates a flaw in the evaluation logic, invalidating the results.
Conclusion
Building reliable AI agents requires more than just a high success rate in a demo. It requires a rigorous evaluation process that preserves the full record of every attempt. Gold Trace Dataworks provides the tools and datasets needed to achieve this level of transparency. By using deterministic grading, traceable lineage, and retained-proof reports, teams can deploy agents with confidence. To start building your evaluation system, send a project request.
