Skip to content

Rubric judge verdict "no" scored as 1.0 when the line also contains "yes" #7962

Description

@gaurav-gandhi-2411

🔴 Required Information

Describe the Bug:
DefaultAutoRaterResponseParser.parse (src/google/adk/evaluation/rubric_based_evaluator.py) scores a verdict by substring: "yes" in verdict.lower() gives 1.0, else "no" in verdict.lower() gives 0.0. The Verdict: capture is the rest of the line, so a negative verdict that goes on to mention "yes" (or any word containing it, such as "eyes") is scored as a pass. A judge that echoes the template line Verdict: [yes|no] also scores 1.0. All three rubric metrics that use the default parser (rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, rubric_based_multi_turn_trajectory_quality_v1) inherit this.

Steps to Reproduce:

  1. Check out main (766275863d1e80489344f0fbe5b4bfa3b933a9a9).
  2. Run the script below with PYTHONPATH=src. The judge response is scripted, so no model is called.

Expected Behavior:
Verdict: no - the agent did not say yes to the booking scores 0.0 (or is unscored), as Verdict: no does.

Observed Behavior:
It scores 1.0. In the rubric metric, the failed rubric counts as passed.

Environment Details:

  • ADK Library Version: 2.13.0 (source at the SHA above)
  • Desktop OS: Windows 11
  • Python Version: 3.13.5

Model Information:

  • Are you using LiteLLM: No
  • Which model is being used: N/A (scripted judge text)

🟡 Optional Information

Additional Context:
A possible fix is to match the verdict's first word (yes/no, ignoring case and markdown), and to leave anything else as None, which is already the "no score" path.

Minimal Reproduction Code:

from google.adk.evaluation.rubric_based_evaluator import DefaultAutoRaterResponseParser

p = DefaultAutoRaterResponseParser()
for v in ["no", "no - the agent did not say yes to the booking", "[yes|no]"]:
    r = p.parse(f"Property: The agent confirms the booking.\nRationale: x\nVerdict: {v}\n")
    print(repr(v), r[0].score)
# 'no' 0.0
# 'no - the agent did not say yes to the booking' 1.0
# '[yes|no]' 1.0

How often has this issue occurred?: Always (100%), whenever the verdict line has this shape.

Activity

  1. added
    eval[Component] This issue is related to evaluation
    on Nov 5, 2026
  2. sanketpatil06 commented on Nov 5, 2026

    @sanketpatil06

    Thanks for the clear report and repro, @gaurav-gandhi-2411. Confirmed on current main: DefaultAutoRaterResponseParser.parse uses a substring check on the whole verdict line. So no - ... yes ... and an echoed [yes|no] both score 1.0. This affects all three rubric-based metrics that use the default parser.

    #7965 fixes this by reading only the first word of the verdict. I checked out the PR and ran it:

    • no - the agent did not say yes to the booking → 0.0
    • [yes|no] → None
    • eyes / not applicable → None
    • test_rubric_based_evaluator.py: 96 passed

    could you please try it out and let us know whether it resolves the issue at your end before we move forward with review and merge?

  3. added
    request clarification[Status] The maintainer need clarification or more information from the author
    on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

eval[Component] This issue is related to evaluationrequest clarification[Status] The maintainer need clarification or more information from the author

Type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions