Score the whole agent run, not just the final text
one move: grade the trajectory
Open the code pane immediately, then use the Spec tab, optional design question, tests, and artifact when you want them.
Finish each move before carrying the artifact forward.
Agent quality is an end-to-end property of tool choices, intermediate evidence, recovery, and final outcome; traces make those steps testable.
Deliverable: State the system invariant in one sentence.Evaluate a trace against expected tools, maximum steps, required evidence, and a final outcome flag.
Deliverable: Type the deterministic core.The evaluator reports each violation, separates tool correctness from outcome success, and is deterministic.
Deliverable: Record a normal case and a boundary case.Insert a wrong tool, a redundant loop, missing evidence, and a successful final answer with an unsafe path.
Deliverable: Name the failure signal and its guard.A trace-grade report with failure categories and a regression fixture.
Deliverable: Save the contract, evidence, and handoff note.The evaluator reports each violation, separates tool correctness from outcome success, and is deterministic.
Insert a wrong tool, a redundant loop, missing evidence, and a successful final answer with an unsafe path.
A trace-grade report with failure categories and a regression fixture.
Deriva-authored extension beyond the supplied PDF. The five moves are Deriva’s implementation contract for this project.