Azure / Azure/AKS

[Feedback] Azure Container Storage with ElasticSAN - csi driver issues during node upgrade

Open
#4,863 4 comments 0 reactions 1 assignee Claimed by @AllenWen-at-Azure View on GitHub
Feedback storage Under Investigation
Dominant language
TypeScript
Stars
2.1k
Forks
395
Avg merge
2d 22h
Merged PRs (30d)
13

Description

**Describe your scenario**
I'm testing Azure Container Storage with ElasticSAN in an AKS cluster (currently in preview). In total, this AKS cluster has 120 PVCs/PVs backed by an ElasticSAN volume across 7 nodes. Everything has been working fine and as expected over the last 4 weeks. However, on 3 or 4 occasions and during a node image upgrade, the pod **azurecontainerstorage-azuresan-csi-driver** failed to start up on the updated node - more precisely, it was the container **driver** on this pod. In the Azure portal, I could see OOMKilled events during this time related to this pod.

**Feedback**
Although I have not been able to fully investigate the issue, I'm not sure if the memory specs for this container are reasonable:
- Even during normal operations, the container **driver** uses up to 100MiB while the request in the specs is only 20MiB. For such a critical pod/container, these numbers might be too low.
- I have no data on memory consumption during the startup phase, but the 512MiB limit might have been too low as well, at least in some instances.
- Unfortunately, I can't reproduce this issue reliably, since during most node upgrades, everything just works fine
- If it happens, the only way to resolve the issue is to drain & delete this node and let the autoscaler create a new one. Restarting the pod does not help.

The last log line visible for the driver container before it was killed were:
```
I0313 07:39:20.594194 158844 main.go:128] "msg"="Configured Azure API client" "logger"="setup" "subscription"="xxxxxxx"
I0313 07:39:20.594224 158844 main.go:153] "msg"="Reconfiguring multipath daemon" "logger"="setup" "numSessions"=32
```

I couldn't find any other issues in the logs of the other containers on this pod.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.