codingjoe / codingjoe/threadmill

Add worker pool telemetry

Open
#18 0 comments 0 reactions 1 assignee Claimed by @codingjoe View on GitHub
enhancement
Dominant language
Python
Stars
12
Forks
1
Avg merge
1d 1h
Merged PRs (30d)
10

Description

We already have queue telemetry, but it would be cool to monitor the health of the worker pool too.

I would love be able to toggle between queue and worker view.

The [psutil](https://github.com/giampaolo/psutil) package allows us to telemetry per node. Via Redis PUB/SUB, we can feed it back to the inspector and display it. I'm thinking of a similar interface like btop or htop (two top alternatives with a nice TUI and graphs).

Some metrics I would be interested in:

node mem
node cpu
node task throughput (tasks per minute)

worker throughput
worker memory allocation
worker CPU allocation

The graphs for throughput, cpu & mem should remain at the same position while displaying different data depending on what is selected.

The selection should be happening via a tree. Queue -> node -> worker
While nodes can repeat per queue, if multiple nodes are listening to the same queue.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.