HarperFast / HarperFast/symphony

Cert hot-reload misses Kubernetes projected-volume (`..data` symlink-swap) rotations

Open
#17 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1
Forks
0
PR merge metrics
No merged PRs in 30d

Description

## Background

`@harperfast/symphony@0.5.0` ([#14](https://github.com/HarperFast/symphony/pull/14)) added live cert-file hot-reload to the standalone `symphony-server`: it watches the parent directories of the cert/key files referenced by the config and, on a change event, filters by the referenced **basename** (`certFilesByDir.get(dir)?.has(filename)`) before triggering a debounced reconcile.

## Limitation

Basename filtering handles **in-place writes and renames of the real files** — which is what host-manager does (writes `keys/fullchain.pem` / `keys/privkey.pem` directly). It does **not** handle rotation performed by **swapping a symlinked directory**, which is how several common environments rotate certs:

- **Kubernetes Secret / projected volumes** (and cert-manager) update a mount atomically by writing a new timestamped dir (`..2026_07_06_...`) and re-pointing the `..data` symlink. The watched directory sees `rename` events for `..data` / `..2026_*`, **none of which match the configured basename** (e.g. `tls.crt`), so they're filtered out and no reconcile fires.
- **Let's Encrypt `live/` symlinks** can hit the same shape depending on how the symlink is replaced.

Net: in a symlink-swap deployment the hot-reload silently no-ops, and symphony keeps serving the old cert until a restart or an unrelated config write. (Per-route `KeyMismatch` isolation and last-good carry-forward from #14 still hold — this is purely about *noticing* the rotation.)

## Possible fixes

1. Resolve symlinks (`fs.realpathSync`) and watch the real target directory, re-resolving on each reconcile.
2. When the watched dir contains a `..data`-style symlink, don't drop unmatched rename events — let them through to the debounced reconcile (it's already coalesced + cheap).
3. A periodic re-stat fallback (compare mtime/inode of referenced files) as a backstop for watchers that miss events.

Option 2 is the smallest; option 1 is the most general. Both should stay bounded (dedup by dir, existing debounce).

## Priority

Low for the current deployment model (host-manager writes real files in place, which works today). Relevant if symphony is ever run under Kubernetes projected volumes / cert-manager.

_Filed as a follow-up to #14. Issue text generated by an LLM (Claude Opus 4.8) on Kris's behalf._

Contributor guide

Open the contributing guide

Research direction

Start by reading the standalone symphony-server hot-reload implementation introduced in #14, focusing on parent-directory watching, basename filtering, and the debounced reconcile. Compare the proposed symlink-swap handling options and verify the chosen behavior against projected-volume rotations and existing direct-file updates. Done means a Kubernetes-style `..data` rotation triggers certificate reload without breaking current filtering or debounce behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, security
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.