liquidmetal-dev / liquidmetal-dev/brigade
M3 follow-up: gRPC-level liveness check in HealthMonitor (TCP probe misses hung handler)
Nobody has claimed this yet.
- Dominant language
- Elixir
- Stars
- 2
- Forks
- 0
- PR merge metrics
- No merged PRs in 30d
Description
Context
M3 gave Brigade dual-liveness for south-edge hosts, but the health probe is a
fast TCP connect only. A flintlockd whose listener is up while its gRPC handler
is deadlocked (or wedged mid-request) still answers the TCP handshake, so the host
reads as healthy and keeps receiving placements it can't actually serve.
Evidence
-
lib/brigade/local_flintlock/health_monitor.ex:65-79—reachable?/1is a
:gen_tcp.connectto the flintlock endpoint:# Fast TCP probe of the flintlock endpoint — a refused/absent listener returns
# immediately, so a slow gRPC dial never stalls the monitor (plan: TCP :9090 ping).The probe closes the socket on connect and returns
true— it never speaks gRPC. -
lib/brigade/local_flintlock/health_monitor.ex:45— result maps straight to
:available/:unreachable.
Scope
- Add a secondary, slower gRPC-level liveness check (e.g. flintlock
ListMicroVMsor a lightweight ping) gated behind the fast TCP probe, so a
stalled gRPC dial never blocks the monitor loop (run it with a bounded timeout /
off the hot path). - Preserve dual-liveness semantics — TCP failure short-circuits as today; the gRPC
check only runs when TCP passes.
Acceptance
- A host whose TCP port is open but whose gRPC handler is dead is flagged unhealthy.
- A healthy host still passes on the fast path (no added latency in the common case).
- The monitor loop never stalls on a slow/hung gRPC dial (bounded timeout).
- Covered by a hermetic test using the fake flintlock (
test/support/fake_flintlock.ex).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in lib/brigade/local_flintlock/health_monitor.ex, especially reachable?/1 and the result mapping around line 45, then inspect test/support/fake_flintlock.ex. Run the existing health-monitor tests first and trace how the fake handles requests. Done means TCP failure still short-circuits, gRPC handler failure marks the host unhealthy, and slow calls cannot stall the monitor loop.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- elixir, grpc
- Domain
- backend, distributed-systems, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 62/100