canonical / canonical/charm-integration-testing
Shared CI storage: rawfile CSI driver mount failure ("Exception calling application: 'img_file'") stalls pods indefinitely
- Dominant language
- Python
- Stars
- 6
- Forks
- 1
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 96
Description
**Note: This issue was generated with AI assistance (GitHub Copilot) based on automated log analysis and triage.**
Filed by @canonical/solutions-qa
## Summary
Shared CI Kubernetes storage infrastructure issue: the `rawfile.csi.openebs.io` CSI driver fails to **mount** a successfully-provisioned PVC, causing the pod to hang in `Pending`/`PodInitializing` indefinitely. This is a different symptom from the already-tracked #330 (provisioning-time `ResourceExhausted: Not enough disk space`) — here PVC provisioning succeeds, but the subsequent mount fails with an internal driver exception.
## Evidence
Test Observer result: test_execution 649012, test_result 12125873 (target: katib-controller rev 1354, neighbor: katib-db-manager rev 1123, backend: mysql-k8s rev 423).
`mysql-k8s-0` pod stuck `Pending`, never progressing past `PodInitializing`:
```
Warning FailedMount 20m kubelet Unable to attach or mount volumes: unmounted volumes=[mysql-k8s-database-7264f1ae], ...: PVC is not bound
Warning FailedMount 10s (x18 over 20m) kubelet MountVolume.MountDevice failed for volume "pvc-6c2bf3f6-71eb-4db9-b9f4-eb1ea3fd4a3c" : rpc error: code = Unknown desc = Exception calling application: 'img_file'
```
PVC itself was successfully provisioned (different from #330's failure mode):
```
Normal ProvisioningSucceeded 22m rawfile.csi.openebs.io_ck-storage-rawfile-csi-node-7sqgq_... Successfully provisioned volume pvc-6c2bf3f6-71eb-4db9-b9f4-eb1ea3fd4a3c
```
Node: `ck-storage-rawfile-csi-node-7sqgq` (different node from #330's `ck-storage-rawfile-csi-node-dvxp5`).
As a result, `katib-db-manager` (which depends on `mysql-k8s` via `relational-db`) never receives database credentials and blocks with `waiting: Waiting for relational-db data`, timing out `test_deploy`.
## Root cause
The `Exception calling application: 'img_file'` error matches a known upstream bug pattern in `openebs/rawfile-localpv` (see [openebs/rawfile-localpv#402](https://github.com/openebs/rawfile-localpv/issues/402)): the CSI driver's per-volume `disk.meta` JSON file is missing/inconsistent for the `img_file` key, so any operation needing that field (mount, resize, etc.) raises an unhandled exception inside the driver instead of surfacing a clear I/O or config error. This points to shared-cluster CSI driver state corruption/inconsistency, not a bug in mysql-k8s or any charm under test.
## Relationship to other issues
Same underlying storage component (`rawfile.csi.openebs.io` on the shared CI Kubernetes cluster) as #330, but a distinct failure mode (mount-time exception vs. provisioning-time disk exhaustion) and a different node — tracking separately for now since the signatures and affected nodes differ, but flagging the relationship in case a shared node-level fix later resolves both.
## Suggested action
This is an infrastructure/environment issue outside of `charm-integration-testing`'s and mysql-k8s's control. Suggest escalating to whoever owns the shared CI Kubernetes cluster's rawfile-CSI storage class to investigate driver/node health (possibly upgrading or restarting the `rawfile-localpv` CSI driver pods, or checking for stale `disk.meta` files as described in the linked upstream issue).
Contributor guide
Research direction
Start with Test Observer execution 649012/result 12125873 and the mysql-k8s pod events, then read openebs/rawfile-localpv#402 and compare the distinct provisioning failure in #330. Confirm that PVC provisioning succeeds but mounting fails with the missing img_file error, identify the affected rawfile CSI node or driver state, and document the escalation to the shared-cluster owner.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- kubernetes, mysql, python
- Domain
- databases, infrastructure, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100