jnovack / jnovack/github-runner-exporter

Unbounded _diag startup replay causes OOM restart loops on long-lived runners

Open
#3 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
3
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Summary

On long-lived self-hosted runners with large `_diag` histories, `github-runner-exporter:v0.2.2` performs an unbounded historical replay during startup. Its resource use grows with retained history, and a failed or interrupted startup begins that work again because replay progress is not durable.

We observed this across a fleet of 20 exporter instances. At the time of investigation:

- 13 instances were continuously cycling.
- 5 instances were failed/start-limited.
- 2 instances with much smaller `_diag` histories were stable.
- Affected `_diag` directories were approximately 2.4-5.1 GiB with 4,700-10,300 files per runner.
- Kernel/cgroup evidence reported `CONSTRAINT_MEMCG`, `OOMKilled=true`, and exit 137.
- Hosts retained substantial available memory, confirming that the failures were isolated to exporter containers rather than host-global OOM.
- Some exporter units had accumulated more than 8,000 restarts.

Runner services themselves remained healthy.

## Version and deployment

- Exporter: `v0.2.2`
- Commit: `e1ed4616534471683ce937dc5cfb43fc2c3769f3`
- Image digest: `sha256:4f3f400918348973ec8e1af59376ecb894b2ca78a88d4e24a19fdcf13a24e249`
- Linux, one exporter container per long-lived runner installation
- Runner directory mounted read-only into the container
- systemd restarts the container on failure

## Reproduction shape

A production-scale reproduction should use a runner directory containing approximately:

- 5 GiB of `_diag` data
- 10,000 `Runner_*.log` and `Worker_*.log` files
- A mixture of completed, rotated, truncated, and currently growing logs

Start v0.2.2 with the runner directory mounted read-only. Observe bytes read, startup duration, peak RSS, resulting series count, and behavior after an interrupted startup or restart.

The exporter scans historical Worker logs during startup. Restarting begins the same replay again because no durable checkpoint is retained. The two stable controls had only about 286-291 MiB of `_diag` history, strongly correlating startup cost with accumulated history.

## Source-level observations

The current implementation appears to make startup cost proportional to retained history:

- Worker logs are recursively enumerated over the replay window.
- Worker files are loaded with `os.ReadFile`, converted to strings, and split into lines.
- Replay pre-seeds historical repo/workflow/job/actor/status label tuples.
- Cursor, harvested-file, counter, histogram, and pending-job state are memory-only.
- There is no durable checkpoint to prevent replaying the same history after restart.
- Same-path truncation/replacement and partial/corrupt oversized-line recovery are not fully represented by a durable file identity and cursor.

This can create repeated full-file allocations and rereads while also permanently allocating historical metric children.

## Expected behavior

Startup and recovery work should be explicitly bounded and observable rather than implicitly determined by total `_diag` size:

- Configurable finite limits for replay age, files, bytes, lines, and duration.
- Streaming parsing rather than whole-file `os.ReadFile`.
- A durable, versioned, checksummed checkpoint containing file identity, offset, partial-line state, current job state, and any persisted counters.
- Atomic checkpoint writes and recovery from a previous valid generation.
- Explicit gap/replay-truncation metrics when limits or corrupt state prevent complete reconstruction.
- Correct handling of rotation, same-path truncation/replacement, deleted files, partial lines, and oversized/corrupt records.
- Resource regression fixtures large enough to represent a 5 GiB/10,000-file runner history.
- Tests demonstrating bounded bytes read and no full history reread after restart.

## Related metric-cardinality concern

The default job metrics use `repo`, `workflow`, `job_name`, and `actor` labels. In our retained Prometheus data this produced tens of thousands of series, with histogram tuples multiplying each workload label combination. The `actor` label can also expose personal data.

A safer future contract would use a low-cardinality operational profile by default, make workload labels explicitly opt-in, enforce a tuple ceiling with drop/overflow telemetry, and avoid historical label pre-seeding.

## Suggested scope

This likely requires a breaking `v0.3` rather than a small patch:

1. Bounded streaming replay.
2. Durable checkpoint and cursor recovery.
3. Rotation/truncation/corruption handling.
4. Low-cardinality default metric profile.
5. Large-history resource tests.
6. Release checksums/provenance and a writable state-volume contract.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing startup replay of historical Worker logs, especially the os.ReadFile-based loading and in-memory replay state described in the issue. Define the checkpoint, bounded streaming, recovery, rotation, and cardinality requirements before changing behavior. Done means bounded replay is observable, restart does not reread all history, and large-history resource tests cover the stated limits and recovery cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus
Domain
observability, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.