aws / aws/containers-roadmap

[service] [request]: EFS CSI driver restarts

Open
#2,142 1 comment 25 reactions 0 assignees View on GitHub
Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**
Investigate EFS CSI driver restarts

**Which service(s) is this request for?**
EKS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
In EFS CSI Driver Addon pods, the efs-plugin container restarts several times before it comes operational.
Kubernetes version `1.26`
Addon version `v1.5.8-eksbuild.1`
I am using the default worker node AMIs.

The restarts eventually stop, and the mounting and Volume creation works well. It's just that this restarts point to some instability

I suspect it restarts because it can't see liveness-probe running. Here are the container logs from `liveness-probe`. There are a lot more lines.
```
W0913 15:04:08.076212 1 connection.go:173] Still connecting to unix:///csi/csi.sock
E0913 15:06:35.272240 1 main.go:74] health check failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0913 15:08:17.266731 1 main.go:64] failed to establish connection to CSI driver: context deadline exceeded
E0913 15:08:21.565558 1 main.go:64] failed to establish connection to CSI driver: context deadline exceeded
```
Got above error message in `liveness-probe` container in both controller and daemon sets pods.

**Are you currently working around this issue?**
no

**Additional context**
Not sure it's important, this is the liveness probe setting. Don't know if it's backwards compatible, but looks like the image version is for `1.27`, not `1.26`
```yaml
- name: liveness-probe
image: >-
602401143452.dkr.ecr.eu-west-2.amazonaws.com/eks/livenessprobe:v2.10.0-eks-1-27-3
```

**Attachments**
```json
{
"name": "efs-csi-controller-d9945fd4f-lkp8v",
"namespace": "kube-system",
"node": "ip-10-99-0-49.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-provisioner",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 0
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-controller-d9945fd4f-ltq8q",
"namespace": "kube-system",
"node": "ip-10-99-2-223.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-provisioner",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 3
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-node-hfg49",
"namespace": "kube-system",
"node": "ip-10-99-0-49.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-driver-registrar",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 1
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-node-l6hpr",
"namespace": "kube-system",
"node": "ip-10-99-2-223.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-driver-registrar",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 6
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the EFS CSI driver behavior on Kubernetes 1.26 with addon v1.5.8-eksbuild.1, reviewing the liveness-probe image setting and the liveness-probe, efs-plugin, controller, and daemon set logs. Done means identifying the cause of the efs-plugin restarts and confirming stable controller and node pods without the reported CSI connection timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, kubernetes
Domain
cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.