Ship warning/error/critical worker logs to Scheduler
- Dominant language
- Python
- Stars
- 1.7k
- Forks
- 778
- Avg merge
- 2h 50m
- Merged PRs (30d)
- 3
Description
Today we depend on the resource manager (like Kubernetes) to track worker logs. We don't offer this ourselves. This is for good reason, we don't want the scheduler to become over-burdened by logs from the workers (which could easily happen).
However, getting worker logs isn't easy for many folks, especially when workers go down. Perhaps we could capture worker logs that are above a certain level of importance, like warning/error/critical in deques. This might give us a good balance.
If we want to do this then I think we would add a `logs: collections.deque(maxlen=1000)` on the scheduler for every worker. We would then also create a logging Handler that scooped up logs above a certain level. We would then either ..
1. Include any update to these in a heartbeat (maybe less prone to saturate the scheduler, but also higher risk of losing the message in a failure case)
2. Send a separate message whenever an exception occurs (more responsive for the scheduler, but maybe also less scalable)
cc @jacobtomlinson
Contributor guide
Assessment
This issue has not been assessed yet.