You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Rubric judge verdict "no" scored as 1.0 when the line also contains "yes" #7962
Describe the Bug: DefaultAutoRaterResponseParser.parse (src/google/adk/evaluation/rubric_based_evaluator.py) scores a verdict by substring: "yes" in verdict.lower() gives 1.0, else "no" in verdict.lower() gives 0.0. The Verdict: capture is the rest of the line, so a negative verdict that goes on to mention "yes" (or any word containing it, such as "eyes") is scored as a pass. A judge that echoes the template line Verdict: [yes|no] also scores 1.0. All three rubric metrics that use the default parser (rubric_based_final_response_quality_v1, rubric_based_tool_use_quality_v1, rubric_based_multi_turn_trajectory_quality_v1) inherit this.
Steps to Reproduce:
Check out main (766275863d1e80489344f0fbe5b4bfa3b933a9a9).
Run the script below with PYTHONPATH=src. The judge response is scripted, so no model is called.
Expected Behavior: Verdict: no - the agent did not say yes to the booking scores 0.0 (or is unscored), as Verdict: no does.
Observed Behavior:
It scores 1.0. In the rubric metric, the failed rubric counts as passed.
Environment Details:
ADK Library Version: 2.13.0 (source at the SHA above)
Desktop OS: Windows 11
Python Version: 3.13.5
Model Information:
Are you using LiteLLM: No
Which model is being used: N/A (scripted judge text)
🟡 Optional Information
Additional Context:
A possible fix is to match the verdict's first word (yes/no, ignoring case and markdown), and to leave anything else as None, which is already the "no score" path.
Minimal Reproduction Code:
fromgoogle.adk.evaluation.rubric_based_evaluatorimportDefaultAutoRaterResponseParserp=DefaultAutoRaterResponseParser()
forvin ["no", "no - the agent did not say yes to the booking", "[yes|no]"]:
r=p.parse(f"Property: The agent confirms the booking.\nRationale: x\nVerdict: {v}\n")
print(repr(v), r[0].score)
# 'no' 0.0# 'no - the agent did not say yes to the booking' 1.0# '[yes|no]' 1.0
How often has this issue occurred?: Always (100%), whenever the verdict line has this shape.
Thanks for the clear report and repro, @gaurav-gandhi-2411. Confirmed on current main: DefaultAutoRaterResponseParser.parse uses a substring check on the whole verdict line. So no - ... yes ... and an echoed [yes|no] both score 1.0. This affects all three rubric-based metrics that use the default parser.
#7965 fixes this by reading only the first word of the verdict. I checked out the PR and ran it:
no - the agent did not say yes to the booking → 0.0
[yes|no] → None
eyes / not applicable → None
test_rubric_based_evaluator.py: 96 passed
could you please try it out and let us know whether it resolves the issue at your end before we move forward with review and merge?
Activity
sanketpatil06 commentedon Nov 5, 2026
Thanks for the clear report and repro, @gaurav-gandhi-2411. Confirmed on current
main:DefaultAutoRaterResponseParser.parseuses a substring check on the whole verdict line. Sono - ... yes ...and an echoed[yes|no]both score 1.0. This affects all three rubric-based metrics that use the default parser.#7965 fixes this by reading only the first word of the verdict. I checked out the PR and ran it:
no - the agent did not say yes to the booking→0.0[yes|no]→Noneeyes/not applicable→Nonetest_rubric_based_evaluator.py: 96 passedcould you please try it out and let us know whether it resolves the issue at your end before we move forward with review and merge?