ObolNetwork / ObolNetwork/obol-stack

Port-conflict fallback is persisted to k3d.yaml and never recovers, permanently moving the stack off :8080

Open
#804 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
11
Forks
1
PR merge metrics
No merged PRs in 30d

Description

Summary

A transient host-port conflict permanently degrades $OBOL_CONFIG_DIR/k3d.yaml. Once obol stack up falls back to ephemeral ingress ports, the stack never returns to 80/8080 — not on restart, not on cluster recreation — even after the conflicting process is long gone. The only recovery is obol stack init --force.

It also silently drops the stack from four ingress mappings to two.

Impact

Every user-facing instruction, doc, and the obol stack up epilogue itself says to visit http://obol.stack:8080. After this fires, that URL is wrong forever and the stack answers on a random high port instead. A user who ran two stacks once — or had anything on port 80 during a single stack up — is left permanently off the documented URL with no obvious way back, because k3d.yaml is not a file most people know to inspect.

Observed in practice: an orphaned older k3d cluster held 80/8080, so a new stack came up on 56854. After deleting the orphan and recreating the cluster, the new cluster still bound 56854 — the ports were free, but the defaults were no longer in the config to be reconsidered.

Root cause

rewriteConflictingPorts (internal/stack/backend_k3d.go:289) is one-directional. It removes conflicting mappings and appends an ephemeral fallback, but never restores a default mapping once the port frees up:

if available(c.hostPort) || owned[c.hostPort] {
    hasMapping[c.containerPort] = true
    continue
}
k3dConfig = strings.Replace(k3dConfig, block, "", 1)   // default mapping deleted

ensureK3dPortsAvailable (internal/stack/backend_k3d.go:439) then persists the result on every stack up, and its own doc comment frames it as handling drift between init and up — but the drift only ever flows one way:

updated := stripConflictingPorts(original, u, false, "")
if updated != original {
    os.WriteFile(configPath, []byte(updated), 0o600)
}

The ratchet:

  1. stack up with 80/8080 occupied → both 80:80 and 8080:80 blocks are deleted; hasMapping[80] is false, so pickPort() appends an ephemeral 56854:80. Same for 443. Persisted.
  2. Ports free up. Next stack up reads a config that now contains only 56854:80. That port is available, so hasMapping[80] = true and nothing changes. The defaults are never reconsidered — they are no longer in the file to be checked.

The 4→2 reduction falls out of the same loop: the template ships 80:80, 8080:80, 443:443, 8443:443, but the recovery path adds at most one mapping per container port.

Reproduction

# occupy the default ingress ports (e.g. a second k3d cluster, or any listener)
obol stack up                     # → "Default ingress ports are in use — use http://obol.stack:56854 instead"
grep -A1 "port:" $OBOL_CONFIG_DIR/k3d.yaml
#   - port: 56854:80
#   - port: 56855:443          # 8080:80 and 8443:443 are gone

# free the ports, then fully recreate the cluster
k3d cluster delete <the-other-cluster>
k3d cluster delete obol-stack-<id>
obol stack up

docker port k3d-obol-stack-<id>-serverlb
#   80/tcp -> 0.0.0.0:56854    # still the fallback, ports 80/8080 are free and unused

Recovery, currently the only way:

obol stack init --force        # preserves the stack ID, regenerates all four mappings
obol stack up

Suggested fix

Make the rewrite reconsider defaults each run instead of mutating a shrinking config. Concretely: derive the port mappings from the embedded template on every up, apply conflict-stripping to that, and treat the on-disk k3d.yaml as derived state rather than the source of truth for ports. Then a freed port is automatically reclaimed on the next stack up.

Two smaller pieces worth fixing alongside:

  • Preserve the secondary mappings — after a conflict the stack should still get both 8080-style and primary mappings where the ports allow, not collapse to one per container port.
  • The fallback warning is printed once, at the moment of the conflict. Since the condition is now sticky, obol stack up should keep surfacing why it is not on :8080 on subsequent runs, and say how to reset.

Notes

  • --force already has the right concept via k3dOwnedHostPorts (a cluster's own ports are not foreign conflicts); the gap is specifically that freed foreign ports are never re-acquired.
  • Found while validating #802 on a live cluster; unrelated to that PR.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in internal/stack/backend_k3d.go, especially rewriteConflictingPorts around line 289 and ensureK3dPortsAvailable around line 439; trace how the embedded template and persisted $OBOL_CONFIG_DIR/k3d.yaml are used during stack up. Reproduce the conflict and recovery sequence with the commands in the issue. Done means freed default ports are reconsidered, all four intended ingress mappings are preserved where available, and subsequent runs explain any remaining fallback.

Written by the indexing model from the issue text.

Assessment

Tech stack
docker, go, kubernetes
Domain
cli, devops, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Clearly specified
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.