dask / dask/distributed

Ship warning/error/critical worker logs to Scheduler

Open
#4,978 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.7k
Forks
778
Avg merge
2h 50m
Merged PRs (30d)
3

Description

Today we depend on the resource manager (like Kubernetes) to track worker logs. We don't offer this ourselves. This is for good reason, we don't want the scheduler to become over-burdened by logs from the workers (which could easily happen).

However, getting worker logs isn't easy for many folks, especially when workers go down. Perhaps we could capture worker logs that are above a certain level of importance, like warning/error/critical in deques. This might give us a good balance.

If we want to do this then I think we would add a `logs: collections.deque(maxlen=1000)` on the scheduler for every worker. We would then also create a logging Handler that scooped up logs above a certain level. We would then either ..

1. Include any update to these in a heartbeat (maybe less prone to saturate the scheduler, but also higher risk of losing the message in a failure case)
2. Send a separate message whenever an exception occurs (more responsive for the scheduler, but maybe also less scalable)

cc @jacobtomlinson

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.