objectstorage-provisioner moving to crashloopbackoff when the cephobjectstore is edited.
- Dominant language
- Go
- Stars
- 56
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
Description:
cephobjectstore has been configured and the objectstorage-provisioner was started and it was running properly.
when I edited the cephobjectstore configuration, objectstore and rgw pod got restarted.
during this time objectstorage-provisioner moved to unknown state and then it got stuck with crashloopbackoff
~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 0/1 Unknown 3 (8m51s ago) 115m pool1-bm08e5
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 0/2 Unknown 2 (8m52s ago) 24m node1
~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 1/1 Running 4 (55s ago) 115m 192.168.0.38 node1
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 1/2 NotReady 4 (55s ago) 24m 192.168.0.37 pool1-bm08e5
~> kubectl get pods -A -o wide | grep obj
rook-ceph objectstorage-controller-7d5c7874c6-wrpr6 1/1 Running 4 (59s ago) 115m 192.168.0.38 node1
rook-ceph objectstorage-provisioner-8bdd4d77d-wthsj 1/2 CrashLoopBackOff 4 (5s ago) 24m 192.168.0.37 node1
Logs:
~> kubectl logs -f objectstorage-provisioner-8bdd4d77d-wthsj -n rook-ceph
Defaulted container "ceph-cosi-driver" out of: ceph-cosi-driver, objectstorage-provisioner-sidecar
E0911 17:45:30.697608 1 server.go:51] "Failed to start server" err="listen unix /var/lib/cosi/cosi.sock: bind: address already in use"
E0911 17:45:30.697681 1 main.go:46] "Exiting on error" err="Failed to start server: listen unix /var/lib/cosi/cosi.sock: bind: address already in use"
Lease:
NAME HOLDER AGE
cosi-ceph-objectstorage-k8s-io-cosi objectstorage-provisioner-8bdd4d77d-wthsj 26m
Workaround:
Manual deletion of the objectstorage-provisioner pod recovers the pod from crashloopbackoff
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with server.go and main.go, where the logs show the COSI socket server startup failure. Reproduce an object store edit while the provisioner is running and inspect the pod and lease state. Done means the provisioner recovers without manual pod deletion and does not enter CrashLoopBackOff because the socket is already in use.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go, kubernetes
- Domain
- infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100