[Bug] Spark blob cleanup test intermittently fails with a missing shuffle ID #10892

Search before asking

• I searched in the issues and found nothing similar.

The related PR discussions and original CI logs are linked below.

Paimon version

2.0 & 2.2-SNAPSHOT.

Compute Engine

Observed in CI with:

• Spark 3.5.8 / Scala 2.12 / JDK 8.
• Spark 3.5.8 / Scala 2.13 / JDK 8.
• Spark 4.1.2 / Scala 2.13 / JDK 17.

Minimal reproduce step

The failure has been observed intermittently in the Spark CI suites containing RemoveOrphanBlobsProcedureTest, in the test Paimon procedure: frozen used mark survives stale sidecar recompute. Three CI occurrences across Spark 3 and Spark 4 are linked below.

There are currently no deterministic reproduction steps for the original CI failure. Running the test alone or rerunning CI may pass.

What doesn't meet your expectations?

Expected: clearShuffleOutputs() skips an ID that was unregistered after enumeration, invalidates the remaining registered shuffles, and propagates other errors.

Actual: it throws InvocationTargetException at line 582 before the deletion-result assertions execute. The cause is SparkException on Spark 3 or ShuffleStatusNotFoundException on Spark 4, reporting that the shuffle ID does not exist.

Excerpt from the PR #10838 CI log, with intermediate frames omitted:

java.lang.reflect.InvocationTargetException:
  ...
  at org.apache.paimon.spark.procedure.RemoveOrphanBlobsProcedureTest.clearShuffleOutputs(RemoveOrphanBlobsProcedureTest.scala:582)
  at org.apache.paimon.spark.procedure.RemoveOrphanBlobsProcedureTest.$anonfun$new$27(RemoveOrphanBlobsProcedureTest.scala:258)
  ...
Cause: org.apache.spark.SparkException: unregisterAllMapAndMergeOutput called for nonexistent shuffle ID 83.
  at org.apache.spark.MapOutputTrackerMaster.unregisterAllMapAndMergeOutput(MapOutputTracker.scala:849)

The helper invalidates shuffle output metadata so that deleted.collect() recomputes after Dataset cache eviction. Removing this invalidation would allow Spark to reuse old shuffle outputs and leave that recomputation path untested.

Anything else?

CI occurrences, with dates in UTC:

• PR [iceberg] Keep tables loadable when the mirror cannot publish their schema #10187, 2026-09-26, Spark 4 / Scala 2.13: the same test fails at clearShuffleOutputs:582 with ShuffleStatusNotFoundException for shuffle ID 82. The affected ScalaTest run reports 1019 succeeded and 1 failed. Original CI log, maintainer discussion.
• PR [core] Skip manual-only tag groups in preview time travel #10284, 2026-09-27, Spark 3 / Scala 2.12: the same test and helper fail with SparkException for shuffle ID 83; 1019 succeeded and 1 failed. Original CI log, maintainer discussion.
• PR [spark] Partition metadata fails on chain and query-authorized reads #10838, 2026-10-29, Spark 3 / Scala 2.13: the same test and helper fail for shuffle ID 83; 1101 succeeded and 1 failed. Original CI log.

The helper was introduced in #9812. Its implementation at the inspected commit enumerates shuffleStatuses and invokes unregisterAllMapAndMergeOutput for each ID:

// Enumeration and invalidation are separate operations.
shuffleStatuses.keys.toSeq.foreach(
  id => unregister.invoke(mapOutputTracker, Int.box(id)))

A concurrent unregisterShuffle(id) can remove the entry between enumeration and invocation. unregisterAllMapAndMergeOutput(id) then looks up the ID again and throws because the entry is absent. This interleaving is consistent with the CI exception, but the logs do not capture the removal or identify the thread responsible.

Proposed fix: catch only the missing-shuffle error for the ID being processed, inside the per-ID loop, and continue with the remaining IDs. Add a deterministic regression test that removes an ID after enumeration and then calls the Paimon helper. Verify that the helper skips the removed ID, still invalidates the remaining registered shuffles, and propagates unrelated errors. Retain the existing stale sidecar recompute and sidecar reads start failing tests.

Are you willing to submit a PR?

• I'm willing to submit a PR!