prometheus / prometheus/node_exporter

expose scheduled shutdown times

Open
#3,110 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
13.8k
Forks
2.7k
Avg merge
1d 23h
Merged PRs (30d)
8

Description

I've been struggling with porting a monitoring check from Nagios to Prometheus. What it does is raise a flag if there's a shutdown scheduled on a server. It does this through this horrendous NRPE check:

command[dsa2_shutdown]=if /usr/lib/nagios/plugins/check_procs -w 1: -u root -C shutdown > /dev/null || /usr/lib/nagios/plugins/check_procs -w 1: -u root -a /lib/systemd/systemd-shutdownd > /dev/null || ( busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown 2> /dev/null | sed 's/[^"]*"//;s/".*//' | grep -v dry- | grep . ); then echo 'system-in-shutdown'; else echo 'no shutdown running' ; fi

i hope you can unsee this one day.

we can probably get rid of all the check_procs stuff there and assume systemd, at least that's what we're asserting it, which turns this into something like:

busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown

and in fact, I wrote a Python script that would extract a metric out of that nicely:

#!/usr/bin/python3

import logging
import shlex
from subprocess import CalledProcessError, PIPE, run


def test_parse_dbus():
    no_sched = '(st) "" 18446744073709551615'
    assert parse_dbus(no_sched) == ("", 0)
    sched_reboot = '(st) "reboot" 1725477267406843'
    assert parse_dbus(sched_reboot) == ("reboot", 1725477267.406843)
    sched_reboot_round = '(st) "reboot" 1725477267506843'
    assert parse_dbus(sched_reboot_round) == ("reboot", 1725477267.506843)
    # theoritical: i've seen the metric "0" with the label "suspend"
    # before adding this test. i couldn't reproduce by suspending my
    # laptop, so i'm not sure wtf happened there.
    sched_suspend = '(st) "suspend" 0'
    assert parse_dbus(sched_suspend) == ("", 0)
    garbage = '(st) "reboot" 1725477267506843 jfdklafjds'
    assert parse_dbus(garbage) == ("", 0)
    assert parse_dbus("(st) ...") == ("", 0)
    assert parse_dbus("") == ("", 0)


def parse_dbus(output: str) -> tuple[str, float]:
    logging.debug("parsing DBus output: %s", output)
    try:
        _, kind, timestamp_str = output.split(maxsplit=2)
    except ValueError as exc:
        logging.warning("could not parse DBus output: %r (%s)", output, exc)
        return "", 0
    kind = kind.replace('"', "")
    try:
        timestamp = int(timestamp_str) / 1000000
    except ValueError as exc:
        logging.warning(
            "could not parse DBus timestamp: %r (%s)",
            timestamp_str,
            exc,
        )
        return "", 0
    logging.debug("found kind %r, timestamp %r", kind, timestamp)
    if kind and timestamp:
        return kind, timestamp
    else:
        return "", 0


def main():
    cmd = shlex.split(
        "busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown"  # noqa: E501
    )
    try:
        proc = run(cmd, check=True, stdout=PIPE, encoding="ascii")
    except CalledProcessError as exc:
        logging.warning("could not call command %r: %s", shlex.join(cmd), exc)
        kind, timestamp = "", 0
    else:
        kind, timestamp = parse_dbus(proc.stdout)
    print("# HELP node_shutdown_scheduled_timestamp_seconds time of the next scheduled reboot, or zero")
    print("# TYPE node_shutdown_scheduled_timestamp_seconds gauge")
    if timestamp:
        print(
            "node_shutdown_scheduled_timestamp_seconds{kind=%s} %s" % (kind, timestamp)
        )
    else:
        print("node_shutdown_scheduled_timestamp_seconds 0")


if __name__ == "__main__":
    main()

the problem is there's nowhere to call this thing from: shutdown(8) doesn't have any post hooks, and i don't think systemd will fire any specific service when a shutdown is scheduled... there are some dbus signal sent around though, namely ScheduledShutdown which we can get with:

busctl get-property org.freedesktop.login1 /org/freedesktop/login1 org.freedesktop.login1.Manager ScheduledShutdown

... which is essentially what we're doing above.

But i figured a better place to do this would be in the node exporter itself, since it's already a daemon just sitting there.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing node_exporter’s existing collector entry points; the issue does not name a file or test. Run the shown busctl command on a systemd host and compare its output with the proposed parser. Done means node_exporter exposes a reliable scheduled-shutdown timestamp metric with the shutdown kind when present.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, linux, python
Domain
observability-sre, operating-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.