hashicorp / hashicorp/nomad

Improve Telemetry for Dispatch Jobs

Open
#4,422 2 comments 0 reactions 0 assignees View on GitHub
theme/metrics type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

### Issue

We use Nomad to dispatch ~many hundreds of thousands~ tens of millions of dispatch jobs.

Currently, the telemetry about these dispatched jobs is extremely poor. There are no summary statistics. The only slightly useful endpoint is `stats.gauges.nomad.nomad.blocked_evals.total_blocked.mbp`, but this isn't particularly useful in a complex system because it isn't broken down by type.

There is a `stats.gauges.nomad.nomad.job_summary.complete` category, but unfortunately it doesn't actually provide any summary statistics, it's just a list of hundreds of thousands of dispatch job names with a value of 0. This is almost worse than useless.

![screen shot 2018-06-15 at 1 26 23 pm](https://user-images.githubusercontent.com/139987/41481672-d905b30e-70a0-11e8-822b-4e3b44546527.png)

It would be **excellent** if there were a `stats.gauges.nomad.nomad.dispatch_summary` where dispatches could be broken down by type so that we could see avg/max usage for cpu/iops/disk/memory for each of our dispatch job types. Without this, there is no useful telemetry for Nomad Dispatch based systems.

Contributor guide

No contributing guide indexed for this repository

Research direction

No files or tests are named in the issue. Start by locating Nomad's existing telemetry for dispatched jobs, including blocked evaluations and job_summary.complete, then trace how dispatch job types and CPU, IOPS, disk, and memory usage are represented. Done means a dispatch_summary telemetry category reports useful average and maximum resource statistics broken down by dispatch job type.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
distributed-systems, observability
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.