spans and annotations lost across copute() boundaries
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
**Describe the issue**:
**Minimal Complete Verifiable Example**:
This is broken for both sync and async delayed functions. Calling functions directly (removing delayed decorator) preserves spans and annotations.
This is an issue for both local and distributed cluster.
```python
import asyncio
import time
from dask import annotate, delayed, get_annotations
from dask.distributed import LocalCluster
from distributed import span
cluster = LocalCluster(processes=False)
client = cluster.get_client()
def outer(i: int):
with span(str(i)), annotate(also_i=i):
print("annotations before compute", get_annotations())
# inner_async(i).compute(optimize_graph=False)
inner_sync(i).compute(optimize_graph=False)
@delayed
def inner_sync(i: int):
for _ in range(5):
time.sleep(1)
print("task", i, "annotations:", get_annotations())
@delayed
async def inner_async(i: int):
for _ in range(5):
await asyncio.sleep(1)
print("task", i, "annotations:", get_annotations())
client.submit(outer, 1)
time.sleep(2)
client.submit(outer, 2).result()
```
Results in:
```python
annotations before compute {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}, 'also_i': 1}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
annotations before compute {'span': {'name': ('2',), 'ids': ('e975dde3-d903-4643-b7ad-ee26f239be78',)}, 'also_i': 2}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 1 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
task 2 annotations: {'span': {'name': ('1',), 'ids': ('e6f9a463-dbea-42d9-86e9-74348f017677',)}}
```
2 things here:
- annotations got lost after entering delayed
- span `2` did not overwrite span `1` for second task
**Anything else we need to know?**:
There is an issue about lost annotations from 2021: https://github.com/dask/dask/issues/7036
It mentions graph optimization but `optimize_graph=False` does not help.
**Environment**:
- Dask version: `2025.11.0`
- Python version: `3.12`
- Operating System: Linux
- Install method (conda, pip, source): uv
Contributor guide
Assessment
This issue has not been assessed yet.