prometheus / prometheus/node_exporter

error gathering metrics: collected metric \"apt_upgrades_pending\" ... was collected before with the same name and label values

Open
#1,462 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

Host operating system:
Linux fra-pg-01 4.4.0-66-generic #87-Ubuntu SMP Fri Mar 3 15:29:05 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux
node_exporter version:
$ /usr/bin/prometheus-node-exporter --version
node_exporter, version 0.17.0+ds (branch: debian/sid, revision: 0.17.0+ds-2)
  build user:       pkg-go-maintainers@lists.alioth.debian.org
  build date:       20181214-14:09:27
  go version:       go1.10.4
node_exporter command line flags
$ cat /etc/default/prometheus-node-exporter
# Set the command-line arguments to pass to the server.
# Due to shell escaping, to pass backslashes for regexes, you need to double
# them (\\d for \d). If running under systemd, you need to double them again
# (\\\\d to mean \d), and escape newlines too.
ARGS="--collector.diskstats.ignored-devices=^(ram|loop|fd)\\\\d+$ \
      --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var/lib/docker/overlay2/[a-z0-9]{64}/merged|var/lib/docker/aufs/mnt/[a-z0-9]{64}|var/lib/docker/devicemapper/mnt/[a-z0-9]{64}|var/lib/docker/containers/[a-z0-9]{64}/shm|var/lib/docker/containers/[a-z0-9]{64}/mounts/shm|var/lib/docker/plugins/[a-z0-9]{64})($|/) \
      --collector.netdev.ignored-devices=^lo$ \
      --collector.textfile.directory=/var/lib/prometheus/node-exporter \
      --collector.ntp \
      --no-collector.hwmon"
Are you running node_exporter in Docker?

No.

What did you do that produced an error?

Didn't do anything particular, just looking at the logs.

What did you expect to see?

No error.

What did you see instead?

The following error:

$ systemctl status prometheus-node-exporter --no-pager -l
● prometheus-node-exporter.service - Prometheus exporter for machine metrics
   Loaded: loaded (/lib/systemd/system/prometheus-node-exporter.service; enabled; vendor preset: enabled)
   Active: active (running) since Thu 2019-05-02 15:32:02 UTC; 3 months 27 days ago
     Docs: https://github.com/prometheus/node_exporter
 Main PID: 2235 (prometheus-node)
    Tasks: 74
   Memory: 25.7M
      CPU: 21h 7min 57.439s
   CGroup: /system.slice/prometheus-node-exporter.service
           └─2235 /usr/bin/prometheus-node-exporter --collector.diskstats.ignored-devices=^(ram|loop|fd)\d+$ --collector.filesystem.ignored-mount-points=^/(sys|proc|dev|run|var/lib/docker/overlay2/[a-z0-9]{64}/merged|var/lib/docker/aufs/mnt/[a-z0-9]{64}|var/lib/docker/devicemapper/mnt/[a-z0-9]{64}|var/lib/docker/containers/[a-z0-9]{64}/shm|var/lib/docker/containers/[a-z0-9]{64}/mounts/shm|var/lib/docker/plugins/[a-z0-9]{64})($|/) --collector.netdev.ignored-devices=^lo$ --collector.textfile.directory=/var/lib/prometheus/node-exporter --collector.ntp --no-collector.hwmon

Aug 29 12:05:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:05:36 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:36Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:05:39 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:05:39Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:06:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:06:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:07:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:07:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:08:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:08:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:09:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:09:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:10:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:10:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:11:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:11:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"
Aug 29 12:12:22 fra-pg-01 prometheus-node-exporter[2235]: time="2019-08-29T12:12:22Z" level=error msg="error gathering metrics: [from Gatherer #2] collected metric \"apt_upgrades_pending\" { label:<name:\"arch\" value:\"Debian/Ubuntu\" > label:<name:\"origin\" value:\"PostgreSQLfor\" > gauge:<value:6 > } was collected before with the same name and label values\n" source="log.go:172"

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the node_exporter metric-gathering path for apt_upgrades_pending and reproduce the duplicate metric using the reported Debian/Ubuntu setup. Trace which collector or input produces the same name and label values, then add coverage for the failure case; done means gathering completes without duplicate-series errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.