aspect-build / aspect-build/aspect-cli
[Bug]: `aspect lint` intermittently SIGSEGVs on static-musl — 12 crashes in 6 days across 2026.35.9 and 2026.36.5
- Dominant language
- Starlark
- Stars
- 165
- Forks
- 45
- Avg merge
- 1d 47m
- Merged PRs (30d)
- 35
Description
### Summary
`aspect lint` dies with SIGSEGV (exit 139) a couple of times a day in our GitHub Actions `lint` job on Aspect Workflows self-hosted runners. **12 occurrences in 6 days**, each on a different runner and a different branch, spanning CLI `2026.35.9` and `2026.36.5`. A plain rerun of the same sha passes.
This is the same class of failure as the one diagnosed in #1355 (`a_crash()` inside mallocng's `enframe`, i.e. an out-of-bounds write) and referenced in #1419 ("a user hit 13 lint-path SIGSEGVs on 2026.31.10"). There is no tracking issue for it, so opening one with our data.
### Environment
- Aspect CLI `2026.35.9` and `2026.36.5`, `x86_64-unknown-linux-musl` (static-pie), stock release binary and the `-debug-` variant.
- Aspect Workflows self-hosted GHA runners (`aw-gha-runner-*`), Linux.
- Command: `aspect lint --task:name lint`.
- Bazel workspace: ~22,800 targets requested, **~950,000 targets configured**, ~4,900 packages, remote cache + BES to BuildBuddy.
### Occurrences
| job started (UTC) | runner | CLI |
|---|---|---|
| 2026-09-03 16:31 | `aw-gha-runner-s7k9` | 2026.35.9 |
| 2026-09-03 20:57 | `aw-gha-runner-4d1n` | 2026.35.9 |
| 2026-09-04 20:26 | `aw-gha-runner-dbkz` | 2026.36.5 |
| 2026-09-04 23:41 | `aw-gha-runner-bhc6` | 2026.36.5 |
| 2026-09-05 00:24 | `aw-gha-runner-2vhp` | 2026.36.5 |
| 2026-09-05 00:56 | `aw-gha-runner-t3x8` | 2026.36.5 |
| 2026-09-06 21:22 | `aw-gha-runner-n65l` | 2026.36.5 |
| 2026-09-07 01:12 | `aw-gha-runner-nkbs` | 2026.36.5 |
| 2026-09-08 17:24 | `aw-gha-runner-gj0t` | 2026.36.5 |
| 2026-09-09 03:35 | `aw-gha-runner-45ls` | 2026.36.5 |
| 2026-09-09 04:10 | `aw-gha-runner-h5sj` | 2026.36.5 |
| 2026-09-09 14:06 | `aw-gha-runner-llcf` | 2026.36.5 (`-debug-`) |
Twelve distinct machines out of a pool that runs this job hundreds of times a day, and one of the crashes was on `main`. Not a sick host, not a branch's content: on a 139 there is no linter output at all, so the diff under test is never implicated.
### What it looks like
```
/.sh: line 1: 50226 Segmentation fault (core dumped) aspect lint --task:name lint
##[error]Process completed with exit code 139.
```
No AXL error, no Rust panic, no allocator message, and (see the companion issue below) no crash report. The CLI dies while `bazel` is still running; the orphaned server keeps writing to the inherited stdio, so the job log shows `Analyzed 22771 targets ... Build completed successfully` *after* the segfault line.
Two clusters, roughly half each:
1. **During analysis** — 6-42 s after the last `Analyzing: N targets (...)` progress line, while lint aspect results are streaming in. The 2026-09-09 14:06 crash is this shape: last `Analyzing:` at 14:06:53, fault at 14:07:35, bazel's own `Build completed successfully` at 14:07:40.
2. **Just after the build finishes** — 1.6-7.1 s after `INFO: Build completed successfully` and the last `Streaming build results to:` line.
Both sit on the build-event stream, not on the shell-out to the linters.
### On the mimalloc A/B from #1355
#1355 made mimalloc the global allocator specifically as an A/B: if the crash survives, the OOB write is in aspect's own code rather than a mallocng quirk. **Every crash above is on a mimalloc build, so our data reads "persists."** Two caveats before you take that as conclusive:
- `#[global_allocator]` only redirects Rust allocations. The binary still links `ring` (0.17.14) and `zstd-sys` (2.0.16), which call libc `malloc`/`free` directly, so mallocng is still live in-process and `enframe`'s check-byte assertion can still be the thing firing.
- On x86_64 musl, `a_crash()` is a bare `hlt` (`arch/x86_64/atomic_arch.h`). It surfaces as SIGSEGV with no message, so in a CI log it is **indistinguishable** from a wild pointer dereference. A mallocng-detected corruption and a genuine bad dereference produce the identical line above.
mimalloc's own detector never fired — `install_allocator_error_handler` aborts on error (exit 134), and we ran with `MIMALLOC_SHOW_ERRORS=1` on the 2026-09-09 14:06 crash with no output. So either the corruption is in the non-mimalloc heap, or it is a plain bad dereference, or mimalloc's heuristic checks simply did not cover it.
### What we ruled out
We audited the full `unsafe` surface of `v2026.36.5`. There is **no raw-buffer `unsafe` on the BEP / named-pipe / broadcaster path** — it is `Vec::resize` + `read_exact` throughout, so the obvious suspect is clean. Three genuine defects turned up, none of them reachable from the lint path, listed here only so they don't get lost:
- `crates/axl-runtime/src/engine/util.rs:11-21` — `freeze_value` transmutes a `Value<'_>` and freezing writes forward headers into a *foreign live heap*. Only reachable from the remote-executor / delivery path (`crates/starbuf-derive/src/lib.rs:2676`).
- `crates/axl-runtime/src/engine/task_context.rs:36-53` — `Defer.kwargs` holds untraced `&'v str` keys.
- `crates/axl-runtime/src/engine/trait.rs:305-311` — `TraitType::export_as` mutates through a `*mut Self` derived from `&self`.
### Suspected amplifier: the lint task subscribes to the whole event stream unfiltered
`crates/aspect-cli/src/builtins/aspect/lint.axl:824` calls `bazel.build_events.iterator()` with **no kind filter**, unlike the shared results iterator which filters through `RESULTS_KINDS` (`private/lib/bazel_results.axl`). On our workspace that means all ~950k `target_configured` events are cloned into an unbounded per-subscriber queue and allocated on a Starlark heap that never collects inside a `def` — the pinned `aspect-forks/starlark-rust` (`bytes_type`, `3470ef71`) only emits GC points at module top level. `private/lib/bazel_results.axl:641` also grows a `target_kinds` dict one entry per label. The drain loop at `lint.axl:902-963` ticks every 250 ms with a ≤10,000-events-per-tick cap, so it runs far behind the producer.
We don't claim that flood *is* the bug — it's safe Rust and safe AXL. But it is a plausible reason this reproduces on our repo and not on smaller ones: it maximizes allocator churn and live-heap size in exactly the window where both crash clusters land. A kind filter on that subscription would be a cheap thing to try independent of the root cause, and would cut a lot of pointless work.
### What would settle it
We have the `-debug-` variant plus `ASPECT_CRASH_LOG` wired into our CI (`actions/upload-artifact` on failure), which is the next step you named in #1419. It produced nothing on the first real crash — filed separately as the companion issue, since that's a handler defect rather than this one.
`(core dumped)` in the shell message means the kernel really is writing a core on these runners. If you can tell us the `core_pattern` / `ulimit -c` setup on Aspect Workflows runners (or ship a knob that keeps the core), we're happy to capture one and hand you a core + the matching unstripped binary. That route doesn't depend on the in-process handler surviving. We can also run any instrumented build you want on this workload — it reproduces reliably at ~2/day and we can point it at a branch build.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with crates/aspect-cli/src/builtins/aspect/lint.axl:824 and the drain loop at lines 902-963, then compare the subscription with private/lib/bazel_results.axl and RESULTS_KINDS. Reproduce `aspect lint --task:name lint` on the stated large workspace using the debug variant and crash artifacts, and investigate the event-stream and allocator behavior. Done means the SIGSEGV is reproduced with actionable evidence or no longer occurs under a validated fix.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, rust
- Domain
- build-system, ci-cd, cli
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100