hashicorp / hashicorp/nomad

e2e: flakey TestMetricsLinux test

Open
#9,496 0 comments 0 reactions 0 assignees View on GitHub
theme/flaky-tests type/bug
Dominant language
Go
Stars
17k
Forks
2.1k
Avg merge
1d 9h
Merged PRs (30d)
105

Description

The [Linux alloc metrics e2e test](https://github.com/hashicorp/nomad/blob/5fbcf82ceefe5facacaff31f45e04ccd89fed3ac/e2e/metrics/metrics.go#L89) occasinally fails with the following error:

```
prometheus query failed (nomad_client_allocs_memory_usage{exported_job="memstress"}): expected non-zero metrics, got: nomad_client_allocs_memory_usage{alloc_id="70ed194c-e5e5-099b-d000-17cc0ad6920e", exported_job="memstress", instance="172.31.13.159:4646", job="nomad_metrics", namespace="default", task="memstress", task_group="memstress"} => 0 @[1606903456.227]
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start in e2e/metrics/metrics.go at line 89 and run the TestMetricsLinux end-to-end test to reproduce the intermittent zero-value metric. Trace the Prometheus query for nomad_client_allocs_memory_usage and make the test reliable; done means the test passes consistently without the reported flaky failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.