bottlerocket-os / bottlerocket-os/bottlerocket
Official support for Spegel
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 11
Description
Related to #1963, #3314
### What I'd like:
I'd like bottlerocket to completely, officially, support Spegel.
### Why
#### Spegel is currently kinda supported
See [comment from Bottlerocket maintainer](https://github.com/bottlerocket-os/bottlerocket/issues/3314#issuecomment-3533851193):
> Indeed, it's the compatibility concern with Spegel that's kept this default in place.
#### Spegel should be supported
Bottlerocket goes hand in hand with karpenter for fast cluster auto-scaling, the exact use-case that Spegel is tailored to.
#### Currently, using Spegel on bottelrocket is painful
See: https://github.com/spegel-org/spegel/issues/47#issuecomment-3547200524
To make Spegel (and alternatives) work on Bottlerocket, we have to resort to workarounds that can break when we upgrade.
The best workaround (writing to `/.bottlerocket/rootfs/etc/containerd/certs.d/_default/hosts.toml`) doesn't work on Bottlerocket ~~for EKS 1.34~~ 1.51. Now we can set `container-registry.mirrors` but we can't configure `capabilities` anymore.
### Alternatives
Personally I use Trow, which works just like Spegel from containerd's perspective, but it can resolve images, so I can do without `capabilities`.
Still, the config required is messy:
```ts
const setupContainerdMirrorScript = outdent`
#!/bin/sh
set -xeuo pipefail
IMDS_TOKEN="$(curl -s -X PUT -H "X-aws-ec2-metadata-token-ttl-seconds: 360" "http://[fd00:ec2::254]/latest/api/token")"
IP="$(curl -H "X-aws-ec2-metadata-token: $IMDS_TOKEN" http://[fd00:ec2::254]/latest/meta-data/${isIpv6Cluster ? "ipv6" : "local-ipv4"})"
apiclient set --json "{\\"container-registry\\":{\\"mirrors\\":{\\"*\\":[\\"http://\${IP}:${TROW_PORT}\\"]}}}"
`;
userDataSections.push(outdent`
[settings.bootstrap-containers.containerd-trow-proxy]
mode = "always"
essential = true
user-data = "${Buffer.from(setupContainerdMirrorScript).toString("base64")}"
`);
```
(the IP thing is necessary on IPv6 clusters as `[::1]:${TROW_PORT}` -same for 127.0.0.1- isn't routed to the Trow service)
Contributor guide
Assessment
This issue has not been assessed yet.