[service] [request]: EFS CSI driver restarts
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
### Community Note
* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment
**Tell us about your request**
Investigate EFS CSI driver restarts
**Which service(s) is this request for?**
EKS
**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**
In EFS CSI Driver Addon pods, the efs-plugin container restarts several times before it comes operational.
Kubernetes version `1.26`
Addon version `v1.5.8-eksbuild.1`
I am using the default worker node AMIs.
The restarts eventually stop, and the mounting and Volume creation works well. It's just that this restarts point to some instability
I suspect it restarts because it can't see liveness-probe running. Here are the container logs from `liveness-probe`. There are a lot more lines.
```
W0913 15:04:08.076212 1 connection.go:173] Still connecting to unix:///csi/csi.sock
E0913 15:06:35.272240 1 main.go:74] health check failed: rpc error: code = DeadlineExceeded desc = context deadline exceeded
E0913 15:08:17.266731 1 main.go:64] failed to establish connection to CSI driver: context deadline exceeded
E0913 15:08:21.565558 1 main.go:64] failed to establish connection to CSI driver: context deadline exceeded
```
Got above error message in `liveness-probe` container in both controller and daemon sets pods.
**Are you currently working around this issue?**
no
**Additional context**
Not sure it's important, this is the liveness probe setting. Don't know if it's backwards compatible, but looks like the image version is for `1.27`, not `1.26`
```yaml
- name: liveness-probe
image: >-
602401143452.dkr.ecr.eu-west-2.amazonaws.com/eks/livenessprobe:v2.10.0-eks-1-27-3
```
**Attachments**
```json
{
"name": "efs-csi-controller-d9945fd4f-lkp8v",
"namespace": "kube-system",
"node": "ip-10-99-0-49.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-provisioner",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 0
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-controller-d9945fd4f-ltq8q",
"namespace": "kube-system",
"node": "ip-10-99-2-223.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-provisioner",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 3
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-node-hfg49",
"namespace": "kube-system",
"node": "ip-10-99-0-49.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-driver-registrar",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 1
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
{
"name": "efs-csi-node-l6hpr",
"namespace": "kube-system",
"node": "ip-10-99-2-223.eu-west-2.compute.internal",
"containers": [
{
"containerName": "csi-driver-registrar",
"restartCount": 0
},
{
"containerName": "efs-plugin",
"restartCount": 6
},
{
"containerName": "liveness-probe",
"restartCount": 0
}
]
}
```
Contributor guide
Research direction
Start by reproducing the EFS CSI driver behavior on Kubernetes 1.26 with addon v1.5.8-eksbuild.1, reviewing the liveness-probe image setting and the liveness-probe, efs-plugin, controller, and daemon set logs. Done means identifying the cause of the efs-plugin restarts and confirming stable controller and node pods without the reported CSI connection timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, kubernetes
- Domain
- cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100