bottlerocket-os / bottlerocket-os/bottlerocket

EKS PersistentVolume accessMode ReadWriteOnce doesn't function correctly

Open
#2,417 4 comments 4 reactions 1 assignee Claimed by @bcressey View on GitHub
area/core area/kubernetes status/icebox type/bug
Dominant language
Rust
Stars
9.7k
Forks
586
Avg merge
1d 11h
Merged PRs (30d)
11

Description

**Image I'm using:**

[bottlerocket-aws-k8s-1.22-x86_64-v1.9.2-b8074d44](https://us-east-1.console.aws.amazon.com/ec2/home?region=us-east-1#Images:visibility=public-images;imageId=ami-04958b57e72e69fb0)

**What I expected to happen:**

According to [PersistentVolume access modes spec](https://kubernetes.io/docs/concepts/storage/persistent-volumes/#access-modes)
> **ReadWriteOnce**
> the volume can be mounted as read-write by a single node. ReadWriteOnce access mode still can **allow multiple pods to access the volume when the pods are running on the same node**.

I expect multiple pods on the same node to be able to read/write to the same PV in parallel (This behavior is implemented correctly in EKS optimized Amazon Linux 2 images).

**What actually happened:**

The volume was mounted successfully to multiple pods on a same node (meaning pods are scheduled and running, in parallel).

Only the first pod can actually read/write the content of the volume, the latter pods get permission denied.

**How to reproduce the problem:**

Apply these manifests on a EKS cluster with[ EBS provisioner](https://github.com/kubernetes-sigs/aws-ebs-csi-driver) installed, or create the PersistentVolume yourself. You must use nodes with bottlerocket as the backing image, ofc!

Observe the logs of 2 pods, one successfully list the content of the volume, the other one get permission denied.

```
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: debug-pvc
spec:
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
storageClassName: ebs-sc
---
apiVersion: v1
kind: Pod
metadata:
name: debug-pod-1
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: debug-pvc
containers:
- name: debug
image: alpine:3
volumeMounts:
- mountPath: /data
name: data
command: ["/bin/sh"]
args: ["-c", "while true; do ls -al /data && echo ----- && sleep 5; done"]
---
apiVersion: v1
kind: Pod
metadata:
name: debug-pod-2
spec:
volumes:
- name: data
persistentVolumeClaim:
claimName: debug-pvc
containers:
- name: debug
image: alpine:3
volumeMounts:
- mountPath: /data
name: data
command: ["/bin/sh"]
args: ["-c", "while true; do ls -al /data && echo ----- && sleep 5; done"]
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.