Azure / Azure/azure-linux-extensions
Linux VM randomly fails to unfreeze disk after taking a Restore Point snapshot
- Dominant language
- Python
- Stars
- 333
- Forks
- 278
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
Recently we started taking Azure Restore Points on our virtual machines as a snapshot to back them up.
Originally, we were stressing the system by taking Restore Points every 5 minutes. The restore point process would be successful only about 50-70% of the time, depending on disk size and which region we were running in.
Therefore our actual restore points were anywhere between 5 to 45 minutes apart as the restore point process would periodically time-out or take significantly longer than expected to capture a restore point.
In this scenario, where we saw high errors and a slowdown of the restore point process, **randomly** the Linux VM's disks would freeze but fail to unfreeze. Non disk-io activity would continue, but otherwise the VM would remain frozen permanently until we deallocated it.
When we extended the restore point snapshot time to 3 hours or so, the restore point snapshots worked reliably again and wouldn't timeout, therefore our VMs didn't have stuck disks again.
We believe this is related to a bug in the usage of safefreeze when used in the Azure VM Agent for Linux.
It appears, but we have not verified, that safefreeze prints to stdout every second before attempting freeze/unfreeze and that output is redirected to a log file.
If restore point snapshot takes too long, potentially the stdout buffer fills and it attempts to write to disk, blocking stdout and blocking the safefreeze application from unfreezing.
```
Linux 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
CentOS Linux release 7.9.2009 (Core)
XFS disks using LVM (to software raid the volume)
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by tracing the Azure VM Agent's safefreeze handling during long Restore Point operations on the reported CentOS 7.9, XFS, and LVM setup. Investigate whether its output or unfreeze path blocks while snapshots run; done means identifying and fixing the condition that leaves disks frozen, then confirming repeated snapshots unfreeze reliably.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, linux, python
- Domain
- cloud, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100