liquidmetal-dev / liquidmetal-dev/brigade

M3 follow-up: gRPC-level liveness check in HealthMonitor (TCP probe misses hung handler)

Open
#17 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
Elixir
Stars
2
Forks
0
PR merge metrics
No merged PRs in 30d

Description

Context

M3 gave Brigade dual-liveness for south-edge hosts, but the health probe is a
fast TCP connect only. A flintlockd whose listener is up while its gRPC handler
is deadlocked (or wedged mid-request) still answers the TCP handshake, so the host
reads as healthy and keeps receiving placements it can't actually serve.

Evidence

  • lib/brigade/local_flintlock/health_monitor.ex:65-79reachable?/1 is a
    :gen_tcp.connect to the flintlock endpoint:

    # Fast TCP probe of the flintlock endpoint — a refused/absent listener returns
    # immediately, so a slow gRPC dial never stalls the monitor (plan: TCP :9090 ping).

    The probe closes the socket on connect and returns true — it never speaks gRPC.

  • lib/brigade/local_flintlock/health_monitor.ex:45 — result maps straight to
    :available / :unreachable.

Scope

  • Add a secondary, slower gRPC-level liveness check (e.g. flintlock
    ListMicroVMs or a lightweight ping) gated behind the fast TCP probe, so a
    stalled gRPC dial never blocks the monitor loop (run it with a bounded timeout /
    off the hot path).
  • Preserve dual-liveness semantics — TCP failure short-circuits as today; the gRPC
    check only runs when TCP passes.

Acceptance

  • A host whose TCP port is open but whose gRPC handler is dead is flagged unhealthy.
  • A healthy host still passes on the fast path (no added latency in the common case).
  • The monitor loop never stalls on a slow/hung gRPC dial (bounded timeout).
  • Covered by a hermetic test using the fake flintlock (test/support/fake_flintlock.ex).

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in lib/brigade/local_flintlock/health_monitor.ex, especially reachable?/1 and the result mapping around line 45, then inspect test/support/fake_flintlock.ex. Run the existing health-monitor tests first and trace how the fake handles requests. Done means TCP failure still short-circuits, gRPC handler failure marks the host unhealthy, and slow calls cannot stall the monitor loop.

Written by the indexing model from the issue text.

Assessment

Tech stack
elixir, grpc
Domain
backend, distributed-systems, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
62/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.