Improve Telemetry for Dispatch Jobs
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
### Issue
We use Nomad to dispatch ~many hundreds of thousands~ tens of millions of dispatch jobs.
Currently, the telemetry about these dispatched jobs is extremely poor. There are no summary statistics. The only slightly useful endpoint is `stats.gauges.nomad.nomad.blocked_evals.total_blocked.mbp`, but this isn't particularly useful in a complex system because it isn't broken down by type.
There is a `stats.gauges.nomad.nomad.job_summary.complete` category, but unfortunately it doesn't actually provide any summary statistics, it's just a list of hundreds of thousands of dispatch job names with a value of 0. This is almost worse than useless.

It would be **excellent** if there were a `stats.gauges.nomad.nomad.dispatch_summary` where dispatches could be broken down by type so that we could see avg/max usage for cpu/iops/disk/memory for each of our dispatch job types. Without this, there is no useful telemetry for Nomad Dispatch based systems.
Contributor guide
No contributing guide indexed for this repository
Research direction
No files or tests are named in the issue. Start by locating Nomad's existing telemetry for dispatched jobs, including blocked evaluations and job_summary.complete, then trace how dispatch job types and CPU, IOPS, disk, and memory usage are represented. Done means a dispatch_summary telemetry category reports useful average and maximum resource statistics broken down by dispatch job type.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- distributed-systems, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100