Spans lifetime management
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
- Follow-up to #7860
Spans currently live forever on the scheduler, and with them the TaskGroups they aggregate.
This meta-data outlasts the Tasks, which are instead dereferenced as soon as they are forgotten, and will slowly increase memory usage on the scheduler.
For clusters that last a few hours, this should not be a problem, as the number of TaskGroups does not scale with the number of tasks: a workflow running on 1 megabyte of data will generate the same amount of permanent metadata as the same algorithm processing 1 petabyte.
In situations where the scheduler is never restarted, this may become a problem.
Ensure that spans are dereferenced after a certain amount of time (if and only if all of their TaskGroups are already been forgotten by the scheduler!), and with them their TaskGroups.
**Note:** TaskGroups are also referenced by Computations, which is deprecated functionality conceptually similar to spans, and that the Spans replaced.
Contributor guide
Assessment
This issue has not been assessed yet.