fosrl / fosrl/newt

Health check config collides between two targets sharing the same backend host:port on reconnect/reload

Open
#411 7 comments 0 reactions 0 assignees View on GitHub
bug needs investigating
Dominant language
Go
Stars
908
Forks
87
Avg merge
1d 15h
Merged PRs (30d)
7

Description

### Describe the Bug

When **two targets point at the same backend `host:port`** but have **different Expected Response Codes**, a websocket **reconnect / config reload** cross-contaminates their health-check configuration: one target inherits the other's expected status code and is then reported **unhealthy forever** even though it responds correctly.

This is distinct from #240 (which reverts custom codes to the 2xx default). Here the code does *not* revert to 2xx — it is overwritten with the *other* target's custom code.

**Setup that triggers it:**
- Target A — expects **200** (e.g. `GET /manifest.json`) → backend `10.0.0.10:8123`
- Target B — expects **401** (e.g. `GET /api/mcp`) → **same backend** `10.0.0.10:8123`

On a **clean start** both load correctly and are healthy. After a websocket drop + reconnect, targets are re-registered (`Replacing existing target with ID ...`) and Target A ends up expecting **401** (Target B's code). Target A then logs `unexpected status code: 200 (expected: 401)` every health-check interval and shows Unhealthy, even though 200 is its correct/expected response.

A clean `newt` restart fixes it (both targets healthy again) — but the next reconnect can re-trigger it.

### Environment

- OS Type & Version: NixOS (newt in a systemd-nspawn container)
- Pangolin Version: 1.20.0 (also observed under 1.19.4)
- Newt Version: 1.14.0
- Gerbil / Traefik / Olm: n/a

### To Reproduce

1. Create two resources whose health-check targets share the **same backend `host:port`**, with **different** Expected Response Codes (e.g. Target A → 200, Target B → 401).
2. Start newt cleanly. Both log `Target X initial status: healthy`.
3. Cause a websocket reconnect (server restart, DNS blip, or brief network drop — enough for newt to log `not connected` then `Websocket connected`).
4. On reconnect newt logs `Replacing existing target with ID A` / `ID B`, then flips one target to `initial status: unhealthy`.
5. That target now logs `Target A: unexpected status code: 200 (expected: 401)` on every interval and stays Unhealthy until newt is restarted.

### Actual log (reconnect race, IDs/IPs genericized)

```
14:42:01 ERROR Failed to send registration message: not connected
... (~35s of "not connected") ...
14:42:34 ERROR Failed to send message: not connected
14:42:37 INFO Websocket connected
14:42:37 INFO Target B initial status: healthy
14:42:37 INFO Target A initial status: healthy <- both healthy right after connect
14:42:37 INFO Replacing existing target with ID A
14:42:37 INFO Replacing existing target with ID B
14:42:37 INFO Target B initial status: healthy
14:42:37 INFO Target A initial status: unhealthy <- flips after the "Replacing" step
14:42:37 WARN Target A: unexpected status code: 200 (expected: 401) <- inherited B's 401
14:43:07 WARN Target A: unexpected status code: 200 (expected: 401)
... repeats every 30s until newt is restarted ...
```

The `initial status: healthy` → `Replacing existing target` → `initial status: unhealthy` sequence for Target A within the same second strongly suggests the re-registration keys health-check config by backend address, so the second target (B) sharing `10.0.0.10:8123` overwrites Target A's expected code.

### Expected Behavior

On reconnect/reload, each target should retain its own configured Expected Response Code regardless of whether another target shares the same backend `host:port`.

### Workaround

Restart newt for a clean reload (both targets load correct codes), or avoid two targets sharing an identical backend `host:port` for health checks (disable the health check on one, or differentiate backends). Both are temporary — the collision returns on the next reconnect.

Contributor guide

Open the contributing guide

Research direction

Start by tracing websocket reconnect and target re-registration, focusing on how health-check configuration is keyed when targets share a backend host:port. Reproduce the issue with two targets expecting different status codes, then verify after reconnect that each target retains its own configured code and remains healthy.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.