bitnami / bitnami/sealed-secrets

Decouple "current key" from "latest key"

Open
#359 2 comments 1 reaction 0 assignees View on GitHub
backlog enhancement
Dominant language
Go
Stars
9.3k
Forks
776
Avg merge
1d 13h
Merged PRs (30d)
30

Description

The suggested backup workflow works a-ok in practice but suffers from a race condition.
It also doesn't resonate with a lot of people (possibly because they intuit such a race condition or possibly just because or doc is bad).

The race condition is the following:

1. An admin (or automation) backs up the current keys
2. Key renewal period (30d) expires and a new key is created
3. Another user seals a secret with `kubeseal` (or fetches a cert with `kubeseal --fetch-cert`)
4. The cluster explodes.

If there is not enough time to run a backup between step 2 and 4, there might be one or more sealed secrets encrypted with a key that hasn't been backed up yet (and thus cannot be recovered after the disaster).

That's not usually a big problem if you run frequent backups (daily) or if you synchronize your backup schedule with the key renewal schedule, but the race condition is still there, albeit for a short amount of time.

----

Sealed secrets works with many keys, and while the youngest key is a bit special because it's the one used by default when sealing a new secret with `kubeseal`, you can seal new secrets by using any of the existing keys as well; that can happen since people can use the "offline" method and pass the certificate explicitly with `--cert`.

The latest (most recent) key is special because it's the most secure (the less likely to have been leaked in the past; simply because it's younger).

Our current default workflow does indeed prioritise security over disaster recoverability.

The subset of users who care about disaster recoverability are quite vocal, so I think we should do something to help them.

A common ask is control over sealing key renewal, but what I really think should happen is control over advertising the public key of the newly created key as "current" (and thus causing the possibility of encrypting an unrecoverable a secret until a backup is done).

A possible "almost zero knob" UX for this would be to add a delay before a new public key is advertised as current.

For example, imagine adding a default delay of renewal period / 2.

Let's draw a period of 10 days and a delay of 5 days:

```
key renew: 1---------2---------3---------4---------5---------
current: -----1---------2---------3---------4---------5----
```

now, this means that as long as you run a periodic backup every 5 days,
no matter what's the phase of your backup w.r.t the key renewal, every time a
key becomes "current", you're guaranteed to have backed it u:

```
key renew: 1---------2---------3---------4---------5---------
current: -----1---------2---------3---------4---------5----
b phase 1: 1----1----2----2----3----3----4----4----5----5----
b phase 2: -1----1----2----2----3----3----4----4----5----5---
b phase 3: --1----1----2----2----3----3----4----4----5----5--
b phase 4: ---1----1----2----2----3----3----4----4----5----5-
b phase 5: ----1----1----2----2----3----3----4----4----5----5
```

i.e. at any vertical slice of that diagram, any of the possible backup schedule "phases" already contains the current key number:

```
key renew: 1---------2---------3---------4---------5---------
current: -----1---------2---------3<

Contributor guide

Open the contributing guide

Research direction

Start by mapping the controller's key-renewal behavior and the kubeseal workflow described in the issue, especially how a newly created public key becomes current. Compare the renewal and backup timing examples to define the required behavior. Done means the current key can be advertised separately from the latest key without allowing newly sealed secrets to depend on an unbacked-up key.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, kubernetes
Domain
security
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.