[EKS Auto Mode] PVC Volume Cloning fails - ec2:CopyVolumes missing from Auto Mode session policy
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
## Issue Overview
PVC volume cloning (using `dataSource` to clone from an existing PVC) fails on EKS Auto Mode clusters because the Auto Mode block storage session policy does not include the `ec2:CopyVolumes` permission.
## Error
```
rpc error: code = Aborted desc = Could not create volume "pvc-xxx": could not create volume in EC2:
operation error EC2: CopyVolumes, https response error StatusCode: 403,
api error UnauthorizedOperation: You are not authorized to perform this operation.
User: arn:aws:sts:::assumed-role/-cluster-role/eks-auto-mode-blockstorage-xxx
is not authorized to perform: ec2:CopyVolumes on resource: arn:aws:ec2:::volume/
because no session policy allows the ec2:CopyVolumes action.
```
## Root Cause
The EKS Auto Mode block storage controller assumes the cluster role using a **scoped session policy**. This session policy does not include `ec2:CopyVolumes`, which is required for EBS volume cloning.
Key observations:
- The `AmazonEKSBlockStoragePolicy` managed policy also does not include `ec2:CopyVolumes`
- Adding `ec2:CopyVolumes` to the cluster role (even with `AdministratorAccess`) does **not** help because the session policy acts as a permissions ceiling
- There is no user-accessible way to modify the Auto Mode session policy
## Steps to Reproduce
1. Create an EKS Auto Mode cluster
2. Create a PVC using `gp3` (or `gp2`) storage class and bind it to a pod
3. Create a second PVC with `dataSource` referencing the first PVC (volume clone)
4. Observe `ProvisioningFailed` with the `ec2:CopyVolumes` unauthorized error
Example PVC clone spec:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: clone-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 15Gi
dataSource:
kind: PersistentVolumeClaim
name: source-pvc
```
## Workarounds Attempted (all failed)
1. **Adding `ec2:CopyVolumes` to the cluster role** — does not work because the session policy ceiling blocks it
2. **Installing self-managed EBS CSI driver addon** — the addon uses provisioner `ebs.csi.aws.com` while Auto Mode uses `ebs.csi.eks.amazonaws.com`, so cross-provisioner cloning fails. Additionally, Auto Mode nodes have a resource-based policy that blocks `ec2:AttachVolume` from non-cluster roles.
3. **Patching DaemonSet node affinity** — the addon's `ebs-csi-node` DaemonSet excludes `compute-type: auto` nodes by default. Even after patching, volume attach still fails due to the resource-based policy on Auto Mode instances.
## Expected Behavior
- `ec2:CopyVolumes` should be included in the Auto Mode block storage session policy
- `AmazonEKSBlockStoragePolicy` should also include `ec2:CopyVolumes` for consistency
- PVC volume cloning should work on EKS Auto Mode clusters
## Impact
- Any application that relies on PVC cloning (database snapshot restore, dev environment provisioning, CI/CD job workspace cloning) is broken on Auto Mode
- No user-side workaround exists — this requires an AWS-side fix to the session policy
## Environment
- EKS Auto Mode, Kubernetes v1.35
- Region: us-east-1
- Storage class: `gp3` / `gp2` with provisioner `ebs.csi.eks.amazonaws.com`
- All nodes: `eks.amazonaws.com/compute-type: auto`
## Related Issues
- #2550 — EBS CSI Driver Incompatibility Between EKS AutoMode and Managed Node Groups
Contributor guide
Research direction
No repository files, tests, or entry points are named. Start by reviewing the EKS Auto Mode block storage session policy and related issue #2550; done means the policy permits ec2:CopyVolumes and the PVC clone reproduction succeeds without an unauthorized-operation error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, databases, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100