prometheus / prometheus/node_exporter

RFE: The systemd collector should have a timeout

Open
#2,585 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

We run our node_exporters with the systemd collector enabled. We've experienced a number of incidents where systemd's dbus interface was either extremely slow or generally non-responsive, and so node_exporter's attempt to collect systemd metrics hangs. At the moment, this stops all metrics collection from node_exporter. It would be nice if the systemd collector had an (optional?) timeout, so that failure to talk to systemd over dbus would merely make it fail instead of causing all host metrics to be unavailable.

The current code in collectors/systemd_linux.go does pass in a context on relevant operations, but the context is context.TODO(). It might be a relatively simple change to add an optional timeout command line argument and then use that to create a different context, although I'm not sure what specifically needs the new context. It might be sufficient to set it up in newSystemdDbusConn() and perhaps leave the other contexts alone.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in collectors/systemd_linux.go, especially newSystemdDbusConn() and the existing context.TODO() calls. Determine which systemd D-Bus operations need the timeout and define how an optional command-line setting should behave; done means a slow or unavailable systemd interface can fail without blocking all node_exporter metrics.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Feature
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.