prometheus / prometheus/node_exporter
RFE: The systemd collector should have a timeout
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 13.8k
- Forks
- 2.7k
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 8
Description
We run our node_exporters with the systemd collector enabled. We've experienced a number of incidents where systemd's dbus interface was either extremely slow or generally non-responsive, and so node_exporter's attempt to collect systemd metrics hangs. At the moment, this stops all metrics collection from node_exporter. It would be nice if the systemd collector had an (optional?) timeout, so that failure to talk to systemd over dbus would merely make it fail instead of causing all host metrics to be unavailable.
The current code in collectors/systemd_linux.go does pass in a context on relevant operations, but the context is context.TODO(). It might be a relatively simple change to add an optional timeout command line argument and then use that to create a different context, although I'm not sure what specifically needs the new context. It might be sufficient to set it up in newSystemdDbusConn() and perhaps leave the other contexts alone.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in collectors/systemd_linux.go, especially newSystemdDbusConn() and the existing context.TODO() calls. Determine which systemd D-Bus operations need the timeout and define how an optional command-line setting should behave; done means a slow or unavailable systemd interface can fail without blocking all node_exporter metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- observability-sre
- Issue type
- Feature
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100