ObolNetwork / ObolNetwork/obol-stack
Port-conflict fallback is persisted to k3d.yaml and never recovers, permanently moving the stack off :8080
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 11
- Forks
- 1
- PR merge metrics
- No merged PRs in 30d
Description
Summary
A transient host-port conflict permanently degrades $OBOL_CONFIG_DIR/k3d.yaml. Once obol stack up falls back to ephemeral ingress ports, the stack never returns to 80/8080 — not on restart, not on cluster recreation — even after the conflicting process is long gone. The only recovery is obol stack init --force.
It also silently drops the stack from four ingress mappings to two.
Impact
Every user-facing instruction, doc, and the obol stack up epilogue itself says to visit http://obol.stack:8080. After this fires, that URL is wrong forever and the stack answers on a random high port instead. A user who ran two stacks once — or had anything on port 80 during a single stack up — is left permanently off the documented URL with no obvious way back, because k3d.yaml is not a file most people know to inspect.
Observed in practice: an orphaned older k3d cluster held 80/8080, so a new stack came up on 56854. After deleting the orphan and recreating the cluster, the new cluster still bound 56854 — the ports were free, but the defaults were no longer in the config to be reconsidered.
Root cause
rewriteConflictingPorts (internal/stack/backend_k3d.go:289) is one-directional. It removes conflicting mappings and appends an ephemeral fallback, but never restores a default mapping once the port frees up:
if available(c.hostPort) || owned[c.hostPort] {
hasMapping[c.containerPort] = true
continue
}
k3dConfig = strings.Replace(k3dConfig, block, "", 1) // default mapping deleted
ensureK3dPortsAvailable (internal/stack/backend_k3d.go:439) then persists the result on every stack up, and its own doc comment frames it as handling drift between init and up — but the drift only ever flows one way:
updated := stripConflictingPorts(original, u, false, "")
if updated != original {
os.WriteFile(configPath, []byte(updated), 0o600)
}
The ratchet:
stack upwith80/8080occupied → both80:80and8080:80blocks are deleted;hasMapping[80]is false, sopickPort()appends an ephemeral56854:80. Same for443. Persisted.- Ports free up. Next
stack upreads a config that now contains only56854:80. That port is available, sohasMapping[80] = trueand nothing changes. The defaults are never reconsidered — they are no longer in the file to be checked.
The 4→2 reduction falls out of the same loop: the template ships 80:80, 8080:80, 443:443, 8443:443, but the recovery path adds at most one mapping per container port.
Reproduction
# occupy the default ingress ports (e.g. a second k3d cluster, or any listener)
obol stack up # → "Default ingress ports are in use — use http://obol.stack:56854 instead"
grep -A1 "port:" $OBOL_CONFIG_DIR/k3d.yaml
# - port: 56854:80
# - port: 56855:443 # 8080:80 and 8443:443 are gone
# free the ports, then fully recreate the cluster
k3d cluster delete <the-other-cluster>
k3d cluster delete obol-stack-<id>
obol stack up
docker port k3d-obol-stack-<id>-serverlb
# 80/tcp -> 0.0.0.0:56854 # still the fallback, ports 80/8080 are free and unused
Recovery, currently the only way:
obol stack init --force # preserves the stack ID, regenerates all four mappings
obol stack up
Suggested fix
Make the rewrite reconsider defaults each run instead of mutating a shrinking config. Concretely: derive the port mappings from the embedded template on every up, apply conflict-stripping to that, and treat the on-disk k3d.yaml as derived state rather than the source of truth for ports. Then a freed port is automatically reclaimed on the next stack up.
Two smaller pieces worth fixing alongside:
- Preserve the secondary mappings — after a conflict the stack should still get both
8080-style and primary mappings where the ports allow, not collapse to one per container port. - The fallback warning is printed once, at the moment of the conflict. Since the condition is now sticky,
obol stack upshould keep surfacing why it is not on:8080on subsequent runs, and say how to reset.
Notes
--forcealready has the right concept viak3dOwnedHostPorts(a cluster's own ports are not foreign conflicts); the gap is specifically that freed foreign ports are never re-acquired.- Found while validating #802 on a live cluster; unrelated to that PR.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in internal/stack/backend_k3d.go, especially rewriteConflictingPorts around line 289 and ensureK3dPortsAvailable around line 439; trace how the embedded template and persisted $OBOL_CONFIG_DIR/k3d.yaml are used during stack up. Reproduce the conflict and recovery sequence with the commands in the issue. Done means freed default ports are reconsidered, all four intended ingress mappings are preserved where available, and subsequent runs explain any remaining fallback.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, kubernetes
- Domain
- cli, devops, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Clearly specified
- Newbie friendliness
- 68/100