alunduil / alunduil/blog.alunduil.com

The observability stack outweighed the bug it was built to catch

未关闭
#619 0 条评论 0 个 reaction 已指派 0 人 在 GitHub 查看
idea
主要语言
Astro
星标
1
派生
1
平均合并
6 小时 34 分钟
30 天内合并 PR
118

描述

## Spark

A long-running workstation-freeze investigation closed this week by deleting the local Alloy/Prometheus/Loki/Tempo/Grafana stack and shipping `sar` output to Grafana Cloud instead.

## Why it could be interesting

The instrumentation built to observe an intermittent freeze was heavy enough to be a suspect in it, so the resolution was to shrink the observer rather than sharpen the observation. The claim: for resource-contention bugs on a single machine, local telemetry infrastructure is a confound before it's a tool, and the right move is to push collection off the box under investigation.

## Open questions

- [ ] Was the freeze diagnosed, or outlived? The post needs a real answer — no claiming a fix that wasn't verified.
- [ ] What did the local stack actually cost in the resources under investigation (memory, IO)? Measured or inferred?
- [ ] Is the Grafana Cloud shipper sufficient to catch a recurrence, and has that been tested?

## Source material

- https://github.com/alunduil/alunduil-chezmoi/issues/404 (closed) and https://github.com/alunduil/alunduil-chezmoi/pull/662 — the swap
- https://github.com/alunduil/alunduil-chezmoi/issues/351 (closed) — retry-on-EAGAIN zellij built to watch the mechanism live
- https://github.com/alunduil/alunduil-chezmoi/issues/347 (closed) — timestamped launch markers
- https://github.com/alunduil/alunduil-chezmoi/issues/356 (closed) — hook cut from ~8 processes to one jq
- https://github.com/alunduil/alunduil-chezmoi/issues/354 (closed) and https://github.com/alunduil/alunduil-chezmoi/pull/671 — least-privilege systemd run-as
- https://github.com/alunduil/alunduil-chezmoi/issues/675 — monitoring moves to system units

贡献指南

这个仓库没有索引到贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。