Dispatched job meta parameters in metrics
- Dominant language
- Go
- Stars
- 17k
- Forks
- 2.1k
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 105
Description
Hello,
I'm using Nomad to handle various tasks such as services & batch jobs and I feel like a feature is missing for batch jobs : I would like to have Meta parameters as Labels in the Alloc metrics API.
### Proposal
In the task runner, there already is a specific case for dispatched jobs : https://github.com/hashicorp/nomad/blob/ff1a30fe8dd4a0da38ddd34915b0be2210d39614/client/allocrunner/taskrunner/task_runner.go#L470-L475
Would it be possible to recursively add meta like
```
Name: "meta_META_NAME",
Value: meta_value
```
?
I don't know if having them by default would be heavy, but we may have a Telemetry parameter to activate this feature (like we already have `publish_allocation_metrics`).
### Use-cases
My Prometheus server is monitoring all my jobs through Nomad Metrics API. I have Grafana dashboards which gives me details about what is happening on the jobs and I have some filters (on `parent_id`, `exported_job`, `alloc_id`, ...) which allows me to select the data I need to see.
I also redacted AlertManager rules to send me alerts when a job fails (and other things).
The purpose would be to be able to filter dispatched jobs based on their meta parameters (and add those details in my alertmanager alerts).
### Attempted Solutions
I activated all telemetry configs, read some articles and some Nomad code to see that the feature doesn't seem to exist yet.
Thank you !
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by reading client/allocrunner/taskrunner/task_runner.go around lines 470-475, then trace how the Alloc metrics API handles labels and how publish_allocation_metrics affects metric publication. Done means dispatched job meta parameters are exposed as metric labels in a way that supports the described Prometheus and Grafana filtering use case.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, grafana, prometheus
- Domain
- api, backend, observability
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100