openshift / openshift/origin

Cannot resize Ceph RBD PVC

Open
#22,789 7 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

lifecycle/frozen
Dominant language
Go
Stars
8.7k
Forks
4.8k
Avg merge
4d 10h
Merged PRs (30d)
53

Description

Hello,

I have an issue with extending pvc based on ceph RBD image. I've created storageclass with allowVolumeExpansion: true parameter to allow pvc expansion for users. However it seems to not working properly with my current setup of OKD. When I set new size for PVC i get following error:

Error expanding volume "xxxx" of plugin kubernetes.io/rbd : rbd info failed, error: can not get image size info kubernetes-dynamic-pvc-3629a9c1-7090-11e9-9aaf-005056ab27db
rbd image 'kubernetes-dynamic-pvc-3629a9c1-7090-11e9-9aaf-005056ab27db':
size 1GiB in 256 objects
order 22 (4MiB objects)
block_name_prefix: rbd_data.a3cc26b8b4567
format: 2
features:
flags:
create_timestamp: Tue May 7 06:20:32 2019

My ceph cluster version is luminous 12.2.12

Version

oc v3.11.0+62803d0-1
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.11.0+d0c29df-98
kubernetes v1.11.0+d4cacc0

Steps To Reproduce
  1. Create pvc
  2. Extend pvc
  3. Wait for pvc to complete resize
Current Result

PVC hangs at "Resizing" and no resize operation is going to happen.

Expected Result

PVC should resize and wait for container to recreate with info:
message: Waiting for user to (re-)start a pod to finish file system resize of volume on node. type: FileSystemResizePending

Additional Information

Upgrading to newest version of 3.11 OKD did not solve the problem.
When trying same operation in my dev environment whole operation succeded and my pvc is expanded. This is older version of OKD:

oc v3.11.0+62803d0-1
kubernetes v1.11.0+d4cacc0
features: Basic-Auth GSSAPI Kerberos SPNEGO

openshift v3.11.0+1c3e643-87
kubernetes v1.11.0+d4cacc0

Can this issue be related to version od OKD? I need this functionality to work with my current version of OKD, but also with newer version in future.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing PVC expansion with Ceph RBD using the listed OKD and Kubernetes versions, then compare the failing setup with the working development environment. Trace the resize operation associated with the reported rbd info failed error; done means the PVC leaves Resizing and reaches the expected filesystem-resize state or completes successfully.

Written by the indexing model from the issue text.

Assessment

Tech stack
kubernetes
Domain
infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.