intel / intel/intel-device-plugins-for-kubernetes
RFC: SGX EPC limits enforcement using NRI to configure misc controller
- Dominant language
- Go
- Stars
- 147
- Forks
- 223
- Avg merge
- 3d 23h
- Merged PRs (30d)
- 18
Description
Since the SGX device plugin (or SGX "support" for Kubernetes in general) was added, we've wanted to be able to enforce per-container `sgx.intel.com/epc: ` values as the hard limits for how much each container gets to use Encrypted Page Cache (EPC) memory but the kernel has not provided the mechanisms for it.
Finally, the Linux kernel has work ongoing to use [misc cgroup controller to limit `sgx_epc` usage](https://lore.kernel.org/linux-sgx/20230923030657.16148-1-haitao.huang@linux.intel.com/T/#mb0dccacb9f4c56d1392b609ad386ec8f0acad3f9). However, this work alone is not sufficient but we also need a mechanism to get `sgx.intel.com/epc: ` values configured to containers' `misc.max`.
The chain is roughly:
```
workload with "sgx.intel.com/epc: 1Mi" -> kubelet -> CRI-O/containerd -> OCI/runc ("sgx_epc 1048 576" > misc.max)
```
To get `"sgx.intel.com/epc: 1Mi"` passed over CRI to CRI-O/containerd, annotations can be used. Most (all?) deployments using the setup from this repository, end up using the SGX mutating webhook that is responsible for pod mutations during `CREATE`. The existing webhook can be modified to add the necessary "EPC limit" values as annotations.
[OCI (Linux) Runtime specification](https://github.com/opencontainers/runtime-spec/blob/main/config-linux.md#control-groups) has limited configuration options to set `misc.max` values:
1. NRI: have an NRI plug-in that registers to 'create container' events and updates `config.json`'s `"unified": { ... }` with the values defined by the annotations. This depends on cgroup-v2 / unified hierarchy to be used by the nodes.
2. [OCI hooks](https://github.com/opencontainers/runtime-spec/blob/main/config.md#createContainer-hooks): have a custom EPC hook that writes to `misc.max` _before_ the container is started. The setup works for both cgroup-v1 and cgroup-v2 but `containerd` does not have proper support to enable OCI hooks. CDI (#1457) can be used but it's in early stages still.
The proposal is to start with 1.
For telemetry/monitoring, `misc.current` can be used to read per-container EPC statistics. For telemetry, the proposal is to start with cAdvisor+Prometheus+Gafana dashboards.
Tasks:
- [ ] integrate with https://github.com/containers/nri-plugins/pull/156: #1582
- [ ] update `pkg/webhooks/sgx` to add NRI annotations: #1582
- [ ] add e2e/sgx test cases
- [ ] (add pod [`ValidatingAdmissionPolicy`](https://kubernetes.io/docs/reference/access-authn-authz/validating-admission-policy/) using CEL language)
- [x] get https://github.com/opencontainers/runc/pull/3972 merged
- [ ] get https://github.com/google/cadvisor/pull/3420 merged
- [ ] prepare cAdvisor deployment that enables `misc stats` and prometheus
- [ ] build Grafana dashboard for demonstrating per-container EPC usage
Future work:
- [ ] https://github.com/kubernetes/enhancements/pull/4113 to get rid of annotations
- [ ] evaluate: SGX NRI plugin to replace SGX device plugin?
Testing:
- Follow this [gist](https://gist.github.com/mythi/0c7381613510a72ed4810d826549290b) to set up the code
Contributor guide
Assessment
This issue has not been assessed yet.