canonical / canonical/charmlibs
Natural reconciler-style use of TLS Certificates API may wipe workload certificates during every renewal
- Dominant language
- Python
- Stars
- 5
- Forks
- 28
- Avg merge
- 1d 5h
- Merged PRs (30d)
- 24
Description
While exploring the TLS Certificates library recently (`charmlibs.interfaces.tls_certificates`), I noticed that reconciler-style usage of the API seems like it could periodically wipe workload certificates during renewal, leading to intermittend TLS outages for workload charms. Agent write-up below (lightly edited).
## Summary
During certificate renewal there is a window in which the library's pull APIs (`get_assigned_certificate()` / `get_assigned_certificates()`) report **no certificates**, even though the previously issued certificate is still valid and still present in the provider's relation data. A reconciler-style charm that derives desired workload state from these APIs on every hook will interpret the empty result as "no certificates desired" and delete the workload's certificate files — causing a brief TLS outage at ~90% of every certificate's lifetime.
## How the gap arises
Renewal is driven by Juju secret expiry. The certificate secret is created with `expire` set at `renewal_relative_time` (default 0.9) of the certificate's validity. When Juju fires `secret_expired`, `_on_secret_expired` calls `_renew_certificate_request`, which — in a single hook — does all of the following:
1. Removes the old CSR from the requirer relation databag (`_remove_requirer_csr_from_relation_data`).
2. Sends a fresh CSR (`_send_certificate_requests`).
3. Removes all revisions of the old certificate secret (`event.secret.remove_all_revisions()`).
From this moment until the provider signs the new CSR and the requirer processes the resulting `relation_changed` (at least one provider hook round-trip, longer if the provider is busy, rate-limited, or temporarily down):
- The old certificate no longer matches any CSR in the requirer databag, so `_find_certificate_in_relation_data` filters it out — the pull APIs return `([], private_key)`.
- The certificate secret is gone, so there is no local copy either.
- No `certificate_available` event fires (it is only emitted for provider certificates matching a *current* CSR).
- The *provider* will additionally delete the now-orphaned certificate entry from its own databag on its next reconcile (`_remove_certificates_for_which_no_csr_exists`).
The workload is unaffected by this if its charm doesn't rewrite its certificates during this time.
## Example
A natural reconciler pattern:
```python
def _reconcile(self, event: ops.EventBase) -> None:
certs, key = self.tls_certificates.get_assigned_certificates()
if not certs:
self._remove_cert_files() # "desired state: no certs"
return
self._push_cert_files(certs, key)
```
observed on `update_status`, other relation events, `config_changed`, scaling events, etc. `update-status` fires periodically, so across a fleet and months of renewals, some unit will reconcile inside the renewal gap. The symptom is a brief, periodic, self-healing TLS outage (cert files disappear, then reappear seconds later when `certificate_available` fires).
## Cause
The pull API conflates three semantically distinct states into one empty
result:
1. **Pending** — CSR outstanding, certificate not yet issued (initial issuance or renewal in flight). Desired action: keep serving current certs.
2. **Absent** — no relation, or no outstanding CSRs. Desired action: tear down.
3. **Revoked** — provider revoked the certificate mid-lifecycle. Desired action: remove promptly.
The information needed to distinguish these exists in the library (`get_csrs_from_requirer_relation_data()`, `get_provider_certificates()` with `revoked` flags, `get_request_errors()`), but nothing in the API surface guides charm authors toward using it.
## Possible remediation options
Agentic analysis suggested some remediation options, but the library maintainers may well have better alternatives.
### Library changes
- **Option A — keep returning the previous certificate during renewal.** `_find_certificate_in_relation_data` could also match provider certificates whose CSR was recently withdrawn but whose certificate still matches the current private key and is not expired/revoked. The old cert is still in the provider databag and still cryptographically valid; filtering it out purely because the CSR was withdrawn is what creates the gap. This makes the naive reconciler safe with no charm-side changes, but is a behaviour change whose potential to cause breakages needs to be considered carefully.
- **Option B — expose state explicitly.** Add a public method or enum, e.g. `get_certificate_state(request) -> PENDING | ASSIGNED | REVOKED | ABSENT`, so reconcilers can branch correctly without hand-rolling the CSR-list and revoked-flag checks.
### Documentation changes
Document the pending/absent/revoked distinction prominently in the requirer usage docs, with a canonical safe reconciler snippet like:
```python
certs, key = self.tls_certificates.get_assigned_certificates()
if certs:
self._push_cert_files(certs, key)
elif not self.model.get_relation(self._relation_name):
self._remove_cert_files() # only on actual departure
# else: issuance/renewal pending — keep serving current certs
```
But this is incomplete, because a reconciler guarded as "empty + relation exists = keep current certs" will keep serving a **revoked** certificate, because revocation is likewise invisible to the pull APIs (the secret is deleted and the provider entry is marked `revoked: true`, so the cert simply vanishes from `get_assigned_certificates()`).
Perhaps a how-to would be warranted, or maybe it's better to document good/bad usage patters in the library docstrings so users and agents have a better chance to see them.
Contributor guide
Research direction
Start by reading the charmlibs.interfaces.tls_certificates flow around _on_secret_expired, _renew_certificate_request, _find_certificate_in_relation_data, get_assigned_certificates(), and the CSR and provider-certificate helpers named in the issue. First reproduce or trace the renewal gap, then clarify which remediation maintainers want; done means the pending, absent, and revoked states are handled without deleting valid workload certificates, with the relevant behavior documented or covered by tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend-api-design, security
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100