hashicorp / hashicorp/nomad

Dispatched job meta parameters in metrics

Open
#14,778 1 comment 0 reactions 0 assignees View on GitHub
stage/needs-discussion theme/metrics type/enhancement
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

Hello,

I'm using Nomad to handle various tasks such as services & batch jobs and I feel like a feature is missing for batch jobs : I would like to have Meta parameters as Labels in the Alloc metrics API.

### Proposal

In the task runner, there already is a specific case for dispatched jobs : https://github.com/hashicorp/nomad/blob/ff1a30fe8dd4a0da38ddd34915b0be2210d39614/client/allocrunner/taskrunner/task_runner.go#L470-L475

Would it be possible to recursively add meta like
```
Name: "meta_META_NAME",
Value: meta_value
```
?

I don't know if having them by default would be heavy, but we may have a Telemetry parameter to activate this feature (like we already have `publish_allocation_metrics`).

### Use-cases

My Prometheus server is monitoring all my jobs through Nomad Metrics API. I have Grafana dashboards which gives me details about what is happening on the jobs and I have some filters (on `parent_id`, `exported_job`, `alloc_id`, ...) which allows me to select the data I need to see.
I also redacted AlertManager rules to send me alerts when a job fails (and other things).

The purpose would be to be able to filter dispatched jobs based on their meta parameters (and add those details in my alertmanager alerts).

### Attempted Solutions

I activated all telemetry configs, read some articles and some Nomad code to see that the feature doesn't seem to exist yet.

Thank you !

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading client/allocrunner/taskrunner/task_runner.go around lines 470-475, then trace how the Alloc metrics API handles labels and how publish_allocation_metrics affects metric publication. Done means dispatched job meta parameters are exposed as metric labels in a way that supports the described Prometheus and Grafana filtering use case.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, grafana, prometheus
Domain
api, backend, observability
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.