dask / dask/distributed

Spans lifetime management

Open
#7,977 1 comment 0 reactions 0 assignees View on GitHub
feature
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

- Follow-up to #7860

Spans currently live forever on the scheduler, and with them the TaskGroups they aggregate.
This meta-data outlasts the Tasks, which are instead dereferenced as soon as they are forgotten, and will slowly increase memory usage on the scheduler.

For clusters that last a few hours, this should not be a problem, as the number of TaskGroups does not scale with the number of tasks: a workflow running on 1 megabyte of data will generate the same amount of permanent metadata as the same algorithm processing 1 petabyte.

In situations where the scheduler is never restarted, this may become a problem.
Ensure that spans are dereferenced after a certain amount of time (if and only if all of their TaskGroups are already been forgotten by the scheduler!), and with them their TaskGroups.

**Note:** TaskGroups are also referenced by Computations, which is deprecated functionality conceptually similar to spans, and that the Spans replaced.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.