Repository navigation
spans and annotations lost across copute() boundaries #9138
Description
Describe the issue:
Minimal Complete Verifiable Example:
This is broken for both sync and async delayed functions. Calling functions directly (removing delayed decorator) preserves spans and annotations.
This is an issue for both local and distributed cluster.
import asyncio
import time
from dask import annotate, delayed, get_annotations
from dask.distributed import LocalCluster
from distributed import span
cluster = LocalCluster(processes=False)
client = cluster.get_client()
def outer(i: int):
with span(str(i)), annotate(also_i=i):
print("annotations before compute", get_annotations())
# inner_async(i).compute(optimize_graph=False)
inner_sync(i).compute(optimize_graph=False)
@delayed
def inner_sync(i: int):
for _ in range(5):
time.sleep(1)
print("task", i, "annotations:", get_annotations())
@delayed
async def inner_async(i: int):
for _ in range(5):
await asyncio.sleep(1)
print("task", i, "annotations:", get_annotations())
client.submit(outer, 1)
time.sleep(2)
client.submit(outer, 2).result()Results in:
annotations before compute {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}, 'also_i': 1}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
annotations before compute {'span': {'name': ('2',), 'ids': ('e975dde3-d903-4643-b7ad-ee26f239be78',)}, 'also_i': 2}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}2 things here:
- annotations got lost after entering delayed
- span
2did not overwrite span1for second task
Anything else we need to know?:
There is an issue about lost annotations from 2021: dask/dask#7036
It mentions graph optimization but optimize_graph=False does not help.
Environment:
- Dask version:
2025.11.0 - Python version:
3.12 - Operating System: Linux
- Install method (conda, pip, source): uv
Activity
Fogapod commentedon Nov 7, 2025
Dont know if
resourcesis special but i tested it as well:this hangs forever because cluster has no
GPU. so its not lost yet before entering.compute()but if i add resources to cluster its same picture with nothing except span
1present in annotations