oracle / oracle/oci-cloud-controller-manager

Vol mount failed - iscsi: error logging in target: exit status 15

Open
#520 0 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
158
Forks
108
PR merge metrics
No merged PRs in 30d

Description

Issue description:
On rebooting node, pod with pvc comes in unknown state and throws following error:

Pod describe o/p:

Events:
  Type     Reason       Age                   From     Message
  ----     ------       ----                  ----     -------
  Warning  FailedMount  4m1s (x659 over 22h)  kubelet  MountVolume.MountDevice failed for volume "csi-b4786769-b306-40ac-9726-dc7aa313b780" : rpc error: code = Internal desc = iscsi: error logging in target: exit status 15

Environment details


CSI release =v1.33.0
K8 version = v1.35.0
OS release = "Ubuntu 24.04.3 LTS"

CSI Node error:

2026-02-06T10:34:54.428Z        ERROR   BV      driver/bv_node.go:192   failed to log into the iSCSI target.    {"volumeID": "ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza", "stagingPath": "/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount", "error": "iscsi: error logging in target: exit status 15"}
2026-02-06T10:34:54.428Z        ERROR   BV      driver/driver.go:352    Failed to process gRPC request. {"error": "rpc error: code = Internal desc = iscsi: error logging in target: exit status 15", "method": "/csi.v1.Node/NodeStageVolume", "request": "{\"publish_context\":{\"attachment-type\":\"iscsi\",\"device\":\"/dev/oracleoci/oraclevdb\",\"iscci_iqn\":\"iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909\",\"iscsi_ip\":\"169.254.2.3\",\"iscsi_port\":\"3260\",\"multipathDevices\":\"\",\"multipathEnabled\":\"false\",\"needResize\":\"false\",\"newSize\":\"\",\"vpusPerGB\":\"10\"},\"staging_target_path\":\"/var/lib/kubelet/plugins/kubernetes.io/csi/blockvolume.csi.oraclecloud.com/325b1e64dcdccc7ca1e75974ac73df635f1b000889017e241b8d8cdd7ba3e7cf/globalmount\",\"volume_capability\":{\"access_mode\":{\"mode\":\"SINGLE_NODE_WRITER\"},\"mount\":{}},\"volume_context\":{\"attachment-type\":\"\",\"needResize\":\"false\",\"newSize\":\"\",\"storage.kubernetes.io/csiProvisionerIdentity\":\"1770228548244-4488-blockvolume.csi.oraclecloud.com\",\"vpusPerGB\":\"10\"},\"volume_id\":\"ocid1.volume.oc1.eu-frankfurt-1.abtheljtbgocla7izk7phwdg3gjimtytpchwyr4chq24ukg7eyhsgca4baza\"}"}
2026-02-06T10:34:57.802Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.803Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.804Z        INFO    FSS     driver/driver.go:354    gRPC response is sent successfully.     {"method": "/csi.v1.Node/NodeGetCapabilities", "response": "{\"capabilities\":[{\"rpc\":{\"type\":\"STAGE_UNSTAGE_VOLUME\"}}]}"}
2026-02-06T10:34:57.806Z        DEBUG   FSS     driver/fss_node.go:62   volumeHandler :  &{ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa 10.0.0.8 /csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8}  {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.806Z        DEBUG   FSS     driver/fss_node.go:78   volume context: map[encryptInTransit:false storage.kubernetes.io/csiProvisionerIdentity:1770275984752-42-fss.csi.oraclecloud.com]       {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z        DEBUG   FSS     driver/fss_node.go:130  Trying to stage.        {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
2026-02-06T10:34:57.816Z        INFO    FSS     driver/fss_node.go:149  Stage started.  {"volumeID": "ocid1.filesystem.oc1.eu_frankfurt_1.aaaaaaaaaandin3ymzzgcllqojxwiotfouwwm4tbnzvwm5lsoqwtcllbmqwtgaaa:10.0.0.8:/csi-fss-083e7b39-9f3d-469f-8f59-68f96c25e5b8"}
--

At OS layer, it seems the device is already logged in however CSI Driver also tried to do and fails

ubuntu@instance-20260202-1239:/var/log$ sudo iscsiadm -m session 
tcp: [1] 169.254.0.2:3260,1 iqn.2015-02.oracle.boot:uefi (non-flash)
tcp: [2] 169.254.2.3:3260,1 iqn.2015-12.com.oracleiaas:274e11de-bd7c-437f-a439-e9b4f6de6909 (non-flash)

If I logout from session using iscsiadm command at OS layer and restart the pod, then volume is mounted successfully. This manual workaround is not feasible for production environment.

csi-oci-node.tar.gz

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with driver/bv_node.go:192 and the NodeStageVolume handling reported through driver/driver.go:352, then inspect the attached csi-oci-node.tar.gz logs. Compare the CSI node's iSCSI login attempt with the existing iscsiadm sessions after reboot. Done means the volume mounts without manually logging out the existing session, with the behavior verified in the reported environment.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.