basecamp / basecamp/kamal-proxy
One unreadable TLS certificate discards the entire saved routing table
- Dominant language
- Go
- Stars
- 1.1k
- Forks
- 89
- Avg merge
- 12h 10m
- Merged PRs (30d)
- 12
Description
## Summary
When kamal-proxy restores its saved state on boot, a single unreadable TLS
certificate causes it to discard **the entire routing table**, not just the
affected service. On a host running several services behind one proxy, one app's
missing cert takes every other app offline at once.
Observed on `basecamp/kamal-proxy:v0.9.2`.
## What happened
Four apps shared one proxy on a single host. One of them was deployed by a
different SSH user than the other three. Since Kamal bind-mounts the proxy's
`apps-config` directory from the deploying user's home:
```
--volume $PWD/.kamal/proxy/apps-config:/home/kamal-proxy/.apps-config
```
that deploy rebooted the proxy with the mount pointing at a different home
directory. The other three apps' certificates were still on disk, but at a path
the proxy could no longer see.
On boot:
```
{"level":"ERROR","msg":"Error loading TLS certificate","error":"open /home/kamal-proxy/.apps-config/app-a/tls/web/cert.pem: no such file or directory"}
{"level":"ERROR","msg":"Failed to decode saved state","path":"/home/kamal-proxy/.config/kamal-proxy/kamal-proxy.state","error":"unable to load certificate"}
{"level":"INFO","msg":"Server started","http":80,"https":443}
```
After that, `kamal-proxy list` returned an empty table and every hostname failed
TLS with `unknown server name` — including the two services whose certificates
were perfectly intact and whose containers were healthy and serving `200` on
`/up` the whole time. Behind Cloudflare this surfaced as HTTP 525 across all
sites simultaneously.
The misconfiguration was ours. What turned it into a multi-app outage is that
one bad cert invalidated the whole state file.
## Expected
A service whose certificate can't be loaded should fail in isolation: log the
error, drop or mark that one service, and restore the remaining services from
the saved state.
## Actual
`Failed to decode saved state` discards all routes. Every service on the host
goes down, and there is no partial recovery — each app has to be re-registered
manually.
## Why it's hard to diagnose
The failure is silent after startup and maximally misleading: the apps are up,
their containers are healthy, their own certificates are fine, and nothing in
their logs indicates a problem. Only the two ERROR lines at proxy boot — which
scroll away — point at the cause.
## Suggested fix
Make state restoration per-service rather than all-or-nothing. Skipping the
services that fail to load, and logging one warning per skipped service, would
have kept this to a single-app outage.
If the all-or-nothing behavior is intentional, it would help a lot to surface it
after boot too — e.g. having `kamal-proxy list` report that a saved state exists
but was rejected, instead of printing an empty table indistinguishable from a
fresh install.
Contributor guide
Research direction
Start with proxy boot's saved-state restoration and the `kamal-proxy list` command, using the reported unreadable-certificate case to trace why one failure empties the table. Compare restored entries for healthy and unreadable services; done means unaffected routes remain available and skipped services produce the requested warning or an explicit state-rejection signal.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100