oxidecomputer / oxidecomputer/omicron
bring back fleet-wide TLS certificates
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
#3095 recently made the TLS certificates silo-scoped for the reasons explained in #2367.
@ahl pointed out a few downsides to this: for a customer who only wants to deal with one TLS certificate (e.g., a wildcard certificate), they have to upload it multiple times (once for each Silo, including the recovery Silo) and also remember to rotate all of the copies. This approach means we also need to provide TLS certificates when creating Silos (#3138) in a real system, even though you generally don't in development (where you're probably using HTTP). This also results in some surprisingly thorny issues with the API and CLI, since the experience we want customers to have (required parameters) is different from what we want to have ourselves (optional parameter).
The reason I was pushing for #3095 was that I didn't want to have to resolve the tricky issues around having both fleet-wide and silo-scoped certificates. But I think this might be simpler than I thought. Proposal:
- there are both silo-scoped and fleet-level certificates
- for any given Silo, if there are any Silo-scoped certificates, we pick from those. otherwise, we pick from the fleet-level certificates
More precisely, here's a summary of how it works today:
- All TLS certificates are silo-scoped.
- The CRUD APIs for certificates are, naturally, silo'ed (under /v1/certificates, and always refers to your Silo's certs).
- Silo Admins can administer them.
- You provide certs at rack initialization. These become the certs for the recovery Silo. It has to work this way since there's no other way to provide them.
- When creating a Silo, you provide the initial certs for the new Silo. If we didn't do this, we'd need to invent new APIs for cross-Silo cert management (which is not a bad idea, just not done).
- To decide what certs to use, Nexus looks at the DNS name you came in on, figures out the Silo, and then picks that Silo's certificate(s).
And here's the proposal to add rack-level certificates:
- A certificate could be silo-scoped or not (internally, silo_id becomes
Option<Uuid>) - /v1/certificates is still CRUD for silo-scoped certificates and you can still do this with Silo Admin privileges.
- We create CRUD APIs at /v1/system/certificates for rack-wide certificates. Fleet admins can administer these.
- You still provide certs at rack initialization just like above. Maybe we let you specify if you want these to be rack-wide or just for the recovery Silo, or maybe for now we just make them rack-level certificates.
- You can still provide certs at silo creation, but you don't have to any more. If you do, they become silo-scoped certs.
- When Nexus decides what certificate to use, it figures out what Silo you're in same as above, and: if that Silo has any silo-scoped certificates, you get that. If it doesn't, you get the rack-wide one.
Pros:
- I think this wouldn't be too much work.
- I think it doesn't raise the nasty questions I was trying to avoid. (I was worried about things like: if Silo A's wildcard cert gets used for Silo B, and Silo A admin wants to delete it, they can break Silo B. In this world, if Silo A uploaded it, Silo B can't use it and so Silo A admin can't break Silo B. If a fleet admin uploaded it, either Silo can use it, and it's the fleet admin who can break everybody.)
- If a customer wants, they can generate one wildcard cert and put it in one place. If they have to rotate it, they still only have to rotate it in one place. If they want separate certs for separate Silos, they can still do this.
- It also solves the problem of required certificates in the "silo create" API. Namely: they're always optional. You could leave out the cert in both dev and prod. In both cases, if Nexus was configured for TLS, you'd get the rack-wide cert (if any). If it wasn't configured for TLS, there's no problem.
The only downside I see with this is that it's some work. I think it'd be good to get this into FCS, but it's not strictly required.
Any thoughts from @ahl @zephraph @rmustacc @askfongjojo or anyone else?
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the current certificate behavior described through #3095, #2367, and #3138, then trace the existing silo-scoped CRUD APIs, rack initialization, silo creation, and Nexus certificate selection. Done means implementing the proposed optional silo scope, fleet-admin /v1/system/certificates CRUD, fallback selection, and preserving silo-scoped certificate behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- api, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100