aws / aws/containers-roadmap

[EKS] [request]: manually-provisioned PersistentVolumes lack topology labels and nodeAffinity

Open
#287 2 comments 4 reactions 0 assignees View on GitHub
EKS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

**Tell us about your request**
Dynamically-provisioned PersistentVolumes using the aws-ebs provisioner are automatically labeled with their zone and region and decorated with the corresponding `nodeAffinity` selectors:

```yaml
apiVersion: v1
kind: PersistentVolume
metadata:
labels:
failure-domain.beta.kubernetes.io/region: us-east-1
failure-domain.beta.kubernetes.io/zone: us-east-1d
spec:
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: failure-domain.beta.kubernetes.io/zone
operator: In
values:
- us-east-1d
- key: failure-domain.beta.kubernetes.io/region
operator: In
values:
- us-east-1
```

I believe that these labels and `nodeAffinity` selectors are required for Kubernetes's multi-zone awareness to work; i.e. they ensure a pod claiming such a volume is scheduled in the correct availability zone. **I would like manually-provisioned PersistentVolumes to be automatically decorated with the same labels and selectors.**

Reading the Kubernetes documentation suggests that this is already supported. In the article about [Well-Known Labels, Annotations, and Taints](https://kubernetes.io/docs/reference/kubernetes-api/labels-annotations-taints/#failure-domainbetakubernetesiozone), it says,

> On the PersistentVolume: The `PersistentVolumeLabel` admission controller will automatically add zone labels to PersistentVolumes, on GCE and AWS.

However, in the [Using Admission Controllers](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#persistentvolumelabel) article, it says,

> PersistentVolumeLabel is DEPRECATED and labeling persistent volumes has been taken over by cloud controller manager. Starting from 1.11, this admission controller is disabled by default.

Further reading about cloud controller manager suggests that it's part of the EKS control plane implementation. I enabled the EKS logs for "controller manager," but I did not see anything in those logs about failure-domain labeling, and my manually-created PVs are not labeled. It appears that neither the admission controller nor cloud controller manager is automatically labeling PersistentVolumes in EBS.

**Which service(s) is this request for?**
EKS running Kubernetes 1.12

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
I want to manually provision PersistentVolumes so that I can manage the lifecycle of the backing EBS volumes myself. In particular, I want to ensure that the data survives and can be re-attached in the event that I need to destroy and re-create the Kubernetes cluster.

**Are you currently working around this issue?**
I am manually applying the labels and `nodeAffinity` selectors as shown in the YAML snippet above. This seems to have allowed my pods to be scheduled and bound to volumes in the correct availability zones. Note the docs also claim that [node affinity](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#node-affinity) is automatically populated for AWS EBS volumes, but manually adding the failure-domain labels was insufficient; I had to manually set node affinity, too.

Contributor guide

Open the contributing guide

Research direction

Start with the Kubernetes PersistentVolume labeling and node-affinity documentation linked in the issue, then compare the shown dynamically provisioned aws-ebs behavior with manually created volumes. Done means EKS automatically applies matching failure-domain labels and nodeAffinity selectors to manually provisioned PersistentVolumes, with multi-zone pod scheduling verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.