aws / aws/containers-roadmap

[EKS Auto Mode] PVC Volume Cloning fails - ec2:CopyVolumes missing from Auto Mode session policy

Open
#2,812 0 comments 0 reactions 0 assignees View on GitHub
EKS EKS Auto Mode
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

## Issue Overview

PVC volume cloning (using `dataSource` to clone from an existing PVC) fails on EKS Auto Mode clusters because the Auto Mode block storage session policy does not include the `ec2:CopyVolumes` permission.

## Error

```
rpc error: code = Aborted desc = Could not create volume "pvc-xxx": could not create volume in EC2:
operation error EC2: CopyVolumes, https response error StatusCode: 403,
api error UnauthorizedOperation: You are not authorized to perform this operation.
User: arn:aws:sts:::assumed-role/-cluster-role/eks-auto-mode-blockstorage-xxx
is not authorized to perform: ec2:CopyVolumes on resource: arn:aws:ec2:::volume/
because no session policy allows the ec2:CopyVolumes action.
```

## Root Cause

The EKS Auto Mode block storage controller assumes the cluster role using a **scoped session policy**. This session policy does not include `ec2:CopyVolumes`, which is required for EBS volume cloning.

Key observations:
- The `AmazonEKSBlockStoragePolicy` managed policy also does not include `ec2:CopyVolumes`
- Adding `ec2:CopyVolumes` to the cluster role (even with `AdministratorAccess`) does **not** help because the session policy acts as a permissions ceiling
- There is no user-accessible way to modify the Auto Mode session policy

## Steps to Reproduce

1. Create an EKS Auto Mode cluster
2. Create a PVC using `gp3` (or `gp2`) storage class and bind it to a pod
3. Create a second PVC with `dataSource` referencing the first PVC (volume clone)
4. Observe `ProvisioningFailed` with the `ec2:CopyVolumes` unauthorized error

Example PVC clone spec:
```yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: clone-pvc
spec:
accessModes:
- ReadWriteOnce
storageClassName: gp3
resources:
requests:
storage: 15Gi
dataSource:
kind: PersistentVolumeClaim
name: source-pvc
```

## Workarounds Attempted (all failed)

1. **Adding `ec2:CopyVolumes` to the cluster role** — does not work because the session policy ceiling blocks it
2. **Installing self-managed EBS CSI driver addon** — the addon uses provisioner `ebs.csi.aws.com` while Auto Mode uses `ebs.csi.eks.amazonaws.com`, so cross-provisioner cloning fails. Additionally, Auto Mode nodes have a resource-based policy that blocks `ec2:AttachVolume` from non-cluster roles.
3. **Patching DaemonSet node affinity** — the addon's `ebs-csi-node` DaemonSet excludes `compute-type: auto` nodes by default. Even after patching, volume attach still fails due to the resource-based policy on Auto Mode instances.

## Expected Behavior

- `ec2:CopyVolumes` should be included in the Auto Mode block storage session policy
- `AmazonEKSBlockStoragePolicy` should also include `ec2:CopyVolumes` for consistency
- PVC volume cloning should work on EKS Auto Mode clusters

## Impact

- Any application that relies on PVC cloning (database snapshot restore, dev environment provisioning, CI/CD job workspace cloning) is broken on Auto Mode
- No user-side workaround exists — this requires an AWS-side fix to the session policy

## Environment

- EKS Auto Mode, Kubernetes v1.35
- Region: us-east-1
- Storage class: `gp3` / `gp2` with provisioner `ebs.csi.eks.amazonaws.com`
- All nodes: `eks.amazonaws.com/compute-type: auto`

## Related Issues

- #2550 — EBS CSI Driver Incompatibility Between EKS AutoMode and Managed Node Groups

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are named. Start by reviewing the EKS Auto Mode block storage session policy and related issue #2550; done means the policy permits ec2:CopyVolumes and the PVC clone reproduction succeeds without an unauthorized-operation error.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, databases, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.