bottlerocket-os / bottlerocket-os/bottlerocket
fsGroupChangePolicy appears to be ignored
- Dominant language
- Rust
- Stars
- 9.7k
- Forks
- 586
- Avg merge
- 1d 18h
- Merged PRs (30d)
- 12
Description
**Image I'm using:**
`ami-0f99e88195df133a4` -- EKS 1.23
**What I expected to happen:**
When setting the following securityContext for a pod, I expected that a recursive chown operation on the attached EBS volume would not happen. Both AL2 & Ubuntu EKS-tuned AMIs appear to properly respect the setting.
* https://kubernetes.io/docs/tasks/configure-pod-container/security-context/#configure-volume-permission-and-ownership-change-policy-for-pods
```
securityContext:
runAsUser: 1000
runAsGroup: 1000
fsGroup: 1000
fsGroupChangePolicy: "OnRootMismatch"
```
**What actually happened:**
All of a pods containers remain stuck in the initializing state until a recursive chown on the attached persistent volume is completed -- at which point the containers will then begin their Initialization. For volumes with a large amount of files, this can take a significant amount of time.
As an example, we run Jenkins on several clusters -- which stores state as lots of small files on disk. Our average deployment has anywhere between 10 - 20 million files on disk which takes 45 minutes or more to complete the chown process when using the BottleRocket AMIs chosen by Karpenter. As stated before, switching the AMI to be Ubuntu or AL2 properly respects the setting and the volume is available for use after ~ 10 seconds.
**How to reproduce the problem:**
Create a PVC that's attached to a Deployment or StatefulSet and generate a million small files on the volume. Forcing a restart of the deployment or statefulset should result in a few minute delay while waiting for the chown to complete.
Contributor guide
Research direction
No source file, test, or entry point is named. Start by reproducing the issue with a PVC containing many files on the Bottlerocket EKS AMI, using fsGroupChangePolicy set to OnRootMismatch, and compare startup behavior with the reported Ubuntu or AL2 result. Done means the volume becomes available without an unnecessary recursive chown.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, linux
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100