ElementsProject / ElementsProject/lightning

CI: ASan/Valgrind shards cancelled mid-run — runner receives shutdown signal (likely OOM on 16GB hosted runner)

Open
#9,279 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
C
Stars
3.1k
Forks
1k
Avg merge
4d 10h
Merged PRs (30d)
13

Description

## Summary

The heaviest CI jobs (`ASan/UBSan` and `Valgrind Test CLN`) intermittently get their runner killed part-way through the test step. The GitHub annotation only shows `Error: The operation was canceled`, with no test failure, which makes it look like "almost every test broke". On the `ASan/UBSan (3/6)` shard it reproduces at almost exactly the 10% mark.

This is not a test failure and not the `concurrency`/`fail-fast` config. The runner VM itself is being terminated mid-run, most likely from memory exhaustion on the 16GB hosted runner.

## Evidence

The real cause is one line above the cancellation in the raw job log (hidden from the annotations view):

```
##[error]The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.
[gw2] [ 10%] PASSED tests/test_closing.py::test_segwit_shutdown_script
##[error]The operation was canceled.
```

No test errored. The runner agent died, so every in-flight and pending test on that shard is reported cancelled.

Sample runs:
- master push: https://github.com/ElementsProject/lightning/actions/runs/28660219314/job/85000852219
- PR #9143: https://github.com/ElementsProject/lightning/actions/runs/28660928500/job/85003365800

## What it is not

- **Not the `concurrency` / `cancel-in-progress` group** (`.github/workflows/ci.yaml:9`). That cancels the whole run. In the master run above only three shards died while every sibling passed:

```
ASan/UBSan (1/6) success ASan/UBSan (3/6) FAILURE ASan/UBSan (4/6) success ...
Valgrind (10/12) FAILURE Valgrind (12/12) FAILURE (other 10 succeeded)
```

- **Not `fail-fast`.** Both `integration-sanitizers` and `integration-valgrind` set `fail-fast: false` (`ci.yaml:729`, `ci.yaml:665`), so a sibling cannot cancel them.
- **Not a timeout.** The shard died at ~6 min; the job timeout is 120 min and the per-test timeout is 1800s.

Individual runner VMs are dying independently, concentrated on exactly the two most memory-heavy job types.

## Likely root cause: memory exhaustion on the 16GB hosted runner

- `runs-on: ubuntu-24.04` is a standard GitHub-hosted runner: 4 vCPU / 16GB RAM.
- The test step runs `pytest -n $(($(nproc) + 1))` -> 5 workers (`ci.yaml:773`; log confirms `created: 5/5 workers`).
- The build under test is `compile-clang-sanitizers`; ASan roughly triples RSS and Valgrind is heavier still.
- At the moment of death, `gw0` had been running `tests/test_askrene.py::test_real_data` (loads the real mainnet gossip map, one of the most memory-hungry tests) for ~3 min alongside four other worker node-clusters.

Five parallel ASan `lightningd`+`bitcoind` clusters with `test_real_data` in the mix peaks past 16GB, and the host force-terminates the VM.

**Why always ~10% on `ASan/UBSan (3/6)`:** `--test-group-random-seed=42` (`ci.yaml:771`) fixes the shard's test set and order deterministically, so the shard hits its memory peak at the same point every run. Other runs hit the same wall on different heavy shards (e.g. `Valgrind 10/12`, `12/12`).

The one thing not provable from the outside is OOM vs. a random hosted-runner reclaim, because GitHub does not expose the VM `dmesg`. The concentration on only ASan/Valgrind plus the deterministic repro point argues strongly for resource exhaustion rather than random reclaim.

## Suggested fixes (cheapest first)

1. Drop the worker count on the sanitizer/valgrind test steps: change `-n $(($(nproc) + 1))` to `-n $(nproc)` or `-n $(($(nproc) - 1))` for those two jobs only. Fewer concurrent node-clusters means lower peak memory.
2. Cap ASan memory via `ASAN_OPTIONS=quarantine_size_mb=64:malloc_context_size=5` in the sanitizer job env.
3. Add an `if: failure()` step running `dmesg | grep -i oom || journalctl -k | tail` so the next failure records the actual cause.

Likely the same underlying overload as the timing-based flakes such as #9268 (an 11s delay to activate `connectd` under load). Related tracking issue: #9222.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.