ClusterLabs / ClusterLabs/resource-agents
Potential Data Loss from lvm_by_vg.sh use of vgremove --removemissing
- Dominant language
- Shell
- Stars
- 519
- Forks
- 608
- Avg merge
- 6d 1h
- Merged PRs (30d)
- 7
Description
I'd like to understand why there are calls in
resource-agents / rgmanager / src / resources / lvm_by_vg.sh
that use vgreduce --removemissing
(Currently at lines 232)
if ! vgreduce --removemissing --force $OCF_RESKEY_vg_name; then
ocf_log err "Failed to make $OCF_RESKEY_vg_name consistent"
return $OCF_ERR_GENERIC
fi
This has caused problems for us with a major client in this scenario:
1) Two clustered servers.
2) New LUNs presented to both servers
3) New LUNs scanned (echo - - - > scsi_hostX/device/scan) on only the ONE service with the resources active
4) Failover initiated because of an incomplete change to the cluster config (although the cause here is irrelevant)
5) Second server cannot 'see' the new LUNs yet and the 'vgremove --removemissing's caused the VG to be "cleaned up" , or imported without the unseen PV's and the LV's contained therein. From this point on, neither server could see the new LV's and the database was unable to start because the VG metadata has been 'broken'.
We were able to recover the situation using vgcfgrestore and archived metadata.
However this is very unexpected behaviour! I would expect the Resource start up to FAIL with an ERROR notice if PV's are missing so that we could fix the problem (simply rescan the scsi_hosts) and _not_ to hose the volume group metadata to the point that it looked initially that we had either major on disc corruption or actual data loss.
So right now this looks like a bug or design fault - and I cannot see a really good reason for the code in question. Please help me understand why its there and consider an alternative approach to making sure the VG can be imported without unexpected side effects.
Many thanks!
Contributor guide
No contributing guide indexed for this repository
Research direction
Read resource-agents/rgmanager/src/resources/lvm_by_vg.sh around line 232 and inspect the vgreduce --removemissing call. Trace the missing-PV failover scenario described, then verify that the resulting behavior preserves VG metadata and reports an error when newly presented LUNs are unseen.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, shell
- Domain
- infrastructure, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100