Feature request: persist evaluation snapshots for auditable score reuse #7988

🔴 Required Information

Is your feature request related to a specific problem?

ADK already separates inference from scoring through BaseEvalService.perform_inference() and evaluate(). I would like to persist a self-contained scoring input snapshot and its evaluation record, so a later process can reuse completed work without silently applying stale scores after reference answers or evaluation rules change.

A concrete workflow:

1. Run an agent against an eval set and score its outputs.
2. Later, inspect or reuse that exact evaluation without calling the agent or judge again.
3. Edit a reference answer or rubric, and score the same agent outputs against the new reference.
4. Keep both evaluations and identify exactly which inputs and rules produced each result.

The current interfaces provide useful building blocks, but callers must assemble the persistence and reuse contract themselves:

• LocalEvalService._evaluate_single_inference_result() looks up the reference case by ID at scoring time.
• Existing results already contain actual/expected invocations and metric configuration. This proposal builds on those records; it does not suggest that ADK only saves a score.
• There is no explicit versioned scoring-input package linking selected inference results, the full reference case/scenario/rubrics, effective configuration, and evaluator implementation identity.
• The built-in judge path converts responses into scores/rationales without a unified persistent record of each rendered request and raw response.
• Manually serializing inference results allows rescoring, but does not establish when a previous score is safe to reuse.

This is a feature proposal based on source inspection, not a claim that every repeated evaluation currently requires rerunning inference.

Describe the Solution You'd Like

Add an opt-in, immutable evaluation snapshot and record with explicit reuse/invalidation semantics.

Input snapshot:

• Selected inference results, embedded or referenced by stable ID and content hash.
• Complete reference case: expected conversation or scenario, plus applicable rubrics.
• Effective metric configuration, including resolved defaults, thresholds, judge settings, and sample count.
• Evaluator, prompt-template, parser, and aggregation identities/versions.
• Snapshot schema version and canonical content fingerprint.

Freeze these inputs before scoring, and score copies rather than modifying persisted inference data. A snapshot must remain usable after the original reference file is edited or deleted; exports must include referenced data.

Evaluation record:

• Snapshot ID, evaluation ID, case/repetition identity, and execution completeness separate from quality status.
• Per-invocation and overall metric results.
• For supported judge evaluators: metric/turn/sample/attempt identity, actual request, raw response or error, parsed score/rationale, and aggregation version.
• An explicit indication when an evaluator cannot provide a complete process trace.

Deterministic evaluators need no judge-call record. Captured requests and responses enable audit and offline parsing/aggregation replay; calling a remote model again with the same prompt does not guarantee identical scores.

Before: current service flow

flowchart TD
    A["Run agent or supply existing inference results"] --> B["InferenceResult objects"]
    B --> C["LocalEvalService.evaluate"]
    D["Read current reference case by ID"] --> C
    E["Metric configuration"] --> C
    C --> F["Execute evaluators and judge calls"]
    F --> G["Yield case results"]
    G --> H["Save final result sets after iteration completes"]
    I["Later scoring request"] --> C
    J["Caller-managed serialization and version tracking"] -.-> B

The current result already preserves useful actual/expected data and metric configuration. The missing capability is a first-class immutable input package and an explicit persisted-score reuse decision.

After: proposed snapshot-based flow

flowchart TD
    A["Select persisted inference results or run agent"] --> B["Freeze reference case and effective scoring rules"]
    B --> C["Persist immutable input snapshot and fingerprint"]
    C --> D{"Explicit reuse requested and complete compatible record exists?"}
    D -->|Yes| E["Return existing evaluation record"]
    D -->|No or force rescore| F["Score the frozen inputs"]
    F --> G["Capture supported judge requests and responses"]
    G --> H["Persist results linked to snapshot"]
    I["Change reference answer, rubric, or scoring rules"] --> B
    J["Change execution input or agent setup"] --> K["Generate new inference results"]
    K --> B

Reuse rules

A valid low score is reusable: quality FAILED differs from execution failure. Conversely, an overall PASSED result alone does not establish completeness when required metrics were not evaluated.

Initially, even threshold changes can conservatively trigger scoring again. Recomputing only verdicts from stored scores should be a later, explicitly supported per-metric optimization.

Impact on your work

We want to compare scoring-rule changes against fixed agent outputs, inspect how scores were produced, and avoid unnecessary agent/judge calls across processes. The goal is trustworthy reuse with visible lineage, rather than a cache that hides changes in evaluation criteria. No fixed deadline.

Willingness to contribute

Yes. I would like to contribute a focused implementation after maintainers confirm the scope and preferred API location. No production-code changes have been made for this proposal.

🟡 Recommended Information

Describe Alternatives You've Considered

• Serialize InferenceResult in application code: useful today, but every application must also manage reference snapshots, defaults, implementation versions, and invalidation.
• Use final result JSON as the snapshot: it contains useful data, but is not a complete pre-scoring input contract or per-sample judge trace.
• Version eval-set files in Git: helps track references, but does not bind inference records to resolved scoring settings and actual judge responses.
• Always rerun everything: avoids stale reuse, but wastes calls and makes comparisons against identical agent behavior harder.

Proposed API / Implementation

API names are intentionally open. A minimal first contribution could provide:

1. Versioned snapshot/record models and optional local storage.
2. Service-level entry points to score a saved snapshot, inspect a result, and explicitly request reuse or force rescoring.
3. Existing inference and metric/result models with additive metadata, rather than a replacement report format.
4. A tracing extension for one built-in judge path, with explicit coverage limitations for other evaluator types.

Implementation constraints:

• Canonicalize content before hashing; preserve sequence ordering and version the canonicalization format.
• A custom evaluator's import path is not an implementation version. Require a trusted version identifier or disable automatic score reuse.
• Restoring a record must not authorize arbitrary executable imports from that record.
• Model aliases and external services can change. Reuse selects a historical result under recorded conditions; it does not prove the external environment is unchanged.
• Publish local records atomically and retain compatibility with existing history files and third-party result managers.
• Capture application data within the configured evaluation storage boundary, exclude credentials, and document redaction/replay limitations.

Out of scope initially: distributed caching, automatic inference deduplication, interrupted-run scheduling/resume, and sample-level checkpoints. Incremental recovery could build on these records later.

Testing plan

• Round-trip snapshots and results across processes.
• Edit/delete the source reference case and verify the historical snapshot is unchanged.
• Verify explicit reuse makes no agent/judge calls for a complete compatible record.
• Change references, rubrics, thresholds, or implementation versions and verify scoring runs again while selected inference outputs remain unchanged.
• Force resampling and retain both records.
• Reject incomplete records as cache hits; allow complete low-quality results.
• Verify scoring does not mutate persisted inference inputs.
• Replay supported judge responses through parsing/aggregation without model calls.
• Verify old result files and existing result managers remain usable.

Additional Context

Source inspected at commit 7662758:

• LocalEvalService
• Result models
• LLM-as-judge execution
• Result storage interface

Related but narrower: #7874 concerns mutation during rescoring; this proposal concerns persisted input identity, audit records, and explicit reuse semantics.

Would maintainers prefer an optional service-level snapshot store, or an additive capability on the existing result-storage abstraction? Is a local-only snapshot/reuse implementation appropriate first, with judge traces following separately?

Investigated and drafted with Codex assistance.