LocalEvalService.evaluate adds rubrics to the caller's inference results, so scoring them again raises #7874

Activity

1. sanketpatil06

   assigned

   surajksharma07

   on Nov 2, 2026
2. sanketpatil06

   added the

   Bug

   issue type on Nov 2, 2026
3. sanketpatil06

   added

   eval[Component] This issue is related to evaluation[Component] This issue is related to evaluation

   on Nov 2, 2026
4. surajksharma07 commented on Nov 5, 2026

   

   surajksharma07

   on Nov 5, 2026

   Collaborator

   More actions

   @adityasingh2400 Reproduced it on current main (2.13.0): the first evaluate passes and the second one fails with “Rubric with rubric_id ‘names_device’ already exists”, because the rubrics end up on the caller’s own invocations.

   Copying the inferences before the rubrics get attached, like you did in #7875, looks like the right direction. Until it lands, passing [r.model_copy(deep=True) for r in inference_results] into each evaluate works as a workaround on my side, so could you try that once on your setup too?

   One case that might still fail: results already saved with rubrics by an older version would still raise on the copy. Could you check that, and maybe add a test that goes through the public evaluate() twice? Then I’ll flag the PR for review.
5. adityasingh2400 commented on Nov 5, 2026

   

   adityasingh2400

   on Nov 5, 2026

   ContributorAuthor

   More actions

   @surajksharma07 Thanks for checking this. You were right about the saved results. Here's what I found on each question.

   Workaround. It works here too, on current main, as long as the results don't carry rubrics yet. With a fresh r.model_copy(deep=True) for each evaluate:

   1st evaluate on a fresh deep copy: PASSED | judged ['confirms_action', 'names_device']
   2nd evaluate on a fresh deep copy: PASSED | judged ['confirms_action', 'names_device']
   the original results still carry [[]]
   

   A deep copy of results that already carry rubrics still raises on main.

   Results saved by an older version. These did still raise on the PR. I ran evaluate and then saved the results on v2.0.0, v2.12.0, v2.13.0 and current main, then scored each file on the branch. All four raised on 823bd23:

   score saved_by_v2.13.0.json (carries [['names_device', 'confirms_action']]): raised ValueError: Rubric with rubric_id 'names_device' already exists.
   

   d3abf17 fixes that. Before attaching rubrics, _drop_rubrics_left_by_earlier_evaluate removes the ones on the copy that are identical to the rubrics about to be attached. All four files now pass:

   score saved_by_v2.13.0.json (carries [['names_device', 'confirms_action']]): PASSED | judged ['confirms_action', 'names_device']
   

   These runs score with rubric_based_final_response_quality_v1 and a fake judge, so the rubrics really do reach the metric, once each.

   Only exact copies are dropped:

   • A rubric with the same id and any changed field (text, description or type) still raises.
   • An id shared by an eval case and one of its turns still raises.
   • A rubric on an inference under any other id stays and gets scored.

   Two calls a reviewer may want to make, both only for files that already carry rubrics:

   • If any field of a rubric changed since the save, it counts as different, so evaluate raises and the run aborts instead of the current version winning.
   • A rubric removed or renamed since the save can't be told apart from one the caller added, so it stays and counts toward the score. On main, that file raised unless none of its rubrics get re-attached.

   Inference results saved after this fix carry no rubrics, so neither case applies to them. The guide already says the eval sets manager must still hold the same cases, so I kept this narrow. Clearing every rubric off the copy would be simpler and would always use the current eval case, but it would also silently drop rubrics a caller set by hand. It's a small change if you'd prefer that.

   Tests. They now go through the public evaluate():

   • test_evaluate_can_score_the_same_inference_results_twice scores the same objects twice. It checks that the caller's results are unchanged and that the metric itself sees the rubrics.
   • test_evaluate_scores_results_saved_with_rubrics_by_an_older_version does a JSON round trip, with rubrics at case level, turn level, both, and a different rubric on each turn.
   • test_evaluate_scores_saved_results_of_a_conversation_scenario covers a case with no expected invocations.
   • test_evaluate_scores_saved_results_when_a_rubric_was_added_since covers a file that holds only some of the rubrics.
   • test_evaluate_keeps_an_inference_rubric_the_eval_case_does_not_add, test_evaluate_keeps_a_rubric_that_belongs_to_another_turn and test_evaluate_still_rejects_a_rubric_id_conflict pin the rules above.

   pytest tests/unittests/evaluation tests/unittests/optimization gives 1226 passed on the branch and 1255 passed merged onto current main. I also tried 22 mutations of the new code, such as removing a guard or matching rubrics by id only, and each one made a test fail.