basecamp / basecamp/kamal-proxy

One unreadable TLS certificate discards the entire saved routing table

Open
#233 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
1.1k
Forks
89
Avg merge
12h 10m
Merged PRs (30d)
12

Description

## Summary

When kamal-proxy restores its saved state on boot, a single unreadable TLS
certificate causes it to discard **the entire routing table**, not just the
affected service. On a host running several services behind one proxy, one app's
missing cert takes every other app offline at once.

Observed on `basecamp/kamal-proxy:v0.9.2`.

## What happened

Four apps shared one proxy on a single host. One of them was deployed by a
different SSH user than the other three. Since Kamal bind-mounts the proxy's
`apps-config` directory from the deploying user's home:

```
--volume $PWD/.kamal/proxy/apps-config:/home/kamal-proxy/.apps-config
```

that deploy rebooted the proxy with the mount pointing at a different home
directory. The other three apps' certificates were still on disk, but at a path
the proxy could no longer see.

On boot:

```
{"level":"ERROR","msg":"Error loading TLS certificate","error":"open /home/kamal-proxy/.apps-config/app-a/tls/web/cert.pem: no such file or directory"}
{"level":"ERROR","msg":"Failed to decode saved state","path":"/home/kamal-proxy/.config/kamal-proxy/kamal-proxy.state","error":"unable to load certificate"}
{"level":"INFO","msg":"Server started","http":80,"https":443}
```

After that, `kamal-proxy list` returned an empty table and every hostname failed
TLS with `unknown server name` — including the two services whose certificates
were perfectly intact and whose containers were healthy and serving `200` on
`/up` the whole time. Behind Cloudflare this surfaced as HTTP 525 across all
sites simultaneously.

The misconfiguration was ours. What turned it into a multi-app outage is that
one bad cert invalidated the whole state file.

## Expected

A service whose certificate can't be loaded should fail in isolation: log the
error, drop or mark that one service, and restore the remaining services from
the saved state.

## Actual

`Failed to decode saved state` discards all routes. Every service on the host
goes down, and there is no partial recovery — each app has to be re-registered
manually.

## Why it's hard to diagnose

The failure is silent after startup and maximally misleading: the apps are up,
their containers are healthy, their own certificates are fine, and nothing in
their logs indicates a problem. Only the two ERROR lines at proxy boot — which
scroll away — point at the cause.

## Suggested fix

Make state restoration per-service rather than all-or-nothing. Skipping the
services that fail to load, and logging one warning per skipped service, would
have kept this to a single-app outage.

If the all-or-nothing behavior is intentional, it would help a lot to surface it
after boot too — e.g. having `kamal-proxy list` report that a saved state exists
but was rejected, instead of printing an empty table indistinguishable from a
fresh install.

Contributor guide

Open the contributing guide

Research direction

Start with proxy boot's saved-state restoration and the `kamal-proxy list` command, using the reported unreadable-certificate case to trace why one failure empties the table. Compare restored entries for healthy and unreadable services; done means unaffected routes remain available and skipped services produce the requested warning or an explicit state-rejection signal.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
50/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.