ClusterLabs / ClusterLabs/anvil
[storage] anvil-manage-server-storage must be able to handle drbd resync during grow
- Dominant language
- Perl
- Stars
- 7
- Forks
- 6
- PR merge metrics
- No merged PRs in 30d
Description
This is not a super common situation, but regardless it needs to be handled properly or storage is leaked during grow processes.
create a server, stop the server to resize root disk (this can happen on any disk, in my test i only had one disk).
Run for the first time:
anvil-manage-server-storage --server an-test-deploy1 --grow 5G --disk vda --confirm
....
Done!
wait for drbd resync to be completed <-- IMPORTANT. All good, you can issue again:
anvil-manage-server-storage --server an-test-deploy1 --grow 5G --disk vda --confirm
....
Done!
and it will work as expected.
wait for drbd resync to be completed <-- IMPORTANT. All good, you can issue:
anvil-manage-server-storage --server an-test-deploy1 --grow 30G --disk vda --confirm
...
Done!
and issue the same command IMMEDIATELY after:
```
# anvil-manage-server-storage --server an-test-deploy1 --grow 30G --disk vda --confirm
Working with the server: [an-test-deploy1], UUID: [d5af3b99-8e57-418f-99d6-90f74372ff78]
- Target: [vda], boot: [01], path: [/dev/drbd/by-res/an-test-deploy1/0], Available space: [130.00 GiB]
- Preparing to grow the storage by: [30.00GiB]...
- Extending local LV: [/dev/anvil-test-vg/an-test-deploy1_0]...
Done!
- Extending peer: [an-a01n02:/dev/anvil-test-vg/an-test-deploy1_0], via: [10.201.10.2 (bcn1)]
Done!
- Extending backing devices complete. Now extending DRBD resource/volume...
Error!
[ Failed ] - When trying to grow the DRBD device: [an-test-deploy1/0]
[ Failed ] - using the command: [/usr/sbin/drbdadm resize an-test-deploy1/0]
[ Failed ] - The return code: [10] was received, expected '0'. Output, if any:
==========
print $output!#
==========
The extension of the resource is incomplete, manual intervention is required!!
[ Note ] - All backing devices have been grown. Manually resolving the drbd grow
[ Note ] - error should complete the drive expansion!
```
This issue is caused by drbd resource refusing a resize one is already in flight. At this point we are leaking storage.
The lv has been resized, but drbd will not see it or recognize it.
Storage is leaked any time a drbd resize request fails, this is just one possible trigger.
For the grow operation specifically, either check drbd status BEFORE resizing the lv and exit 1 if in progress (avoid leaking) or a loop is necessary to wait for the first sync to complete before issuing the next resize.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.