Azure / Azure/azure-linux-extensions

Linux VM randomly fails to unfreeze disk after taking a Restore Point snapshot

Open
#1,680 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
333
Forks
278
Avg merge
2d 9h
Merged PRs (30d)
4

Description

Recently we started taking Azure Restore Points on our virtual machines as a snapshot to back them up.

Originally, we were stressing the system by taking Restore Points every 5 minutes. The restore point process would be successful only about 50-70% of the time, depending on disk size and which region we were running in.

Therefore our actual restore points were anywhere between 5 to 45 minutes apart as the restore point process would periodically time-out or take significantly longer than expected to capture a restore point.

In this scenario, where we saw high errors and a slowdown of the restore point process, **randomly** the Linux VM's disks would freeze but fail to unfreeze. Non disk-io activity would continue, but otherwise the VM would remain frozen permanently until we deallocated it.

When we extended the restore point snapshot time to 3 hours or so, the restore point snapshots worked reliably again and wouldn't timeout, therefore our VMs didn't have stuck disks again.

We believe this is related to a bug in the usage of safefreeze when used in the Azure VM Agent for Linux.

It appears, but we have not verified, that safefreeze prints to stdout every second before attempting freeze/unfreeze and that output is redirected to a log file.

If restore point snapshot takes too long, potentially the stdout buffer fills and it attempts to write to disk, blocking stdout and blocking the safefreeze application from unfreezing.

```
Linux 3.10.0-1160.83.1.el7.x86_64 #1 SMP Wed Jan 25 16:41:43 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

CentOS Linux release 7.9.2009 (Core)

XFS disks using LVM (to software raid the volume)
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Azure VM Agent's safefreeze handling during long Restore Point operations on the reported CentOS 7.9, XFS, and LVM setup. Investigate whether its output or unfreeze path blocks while snapshots run; done means identifying and fixing the condition that leaves disks frozen, then confirming repeated snapshots unfreeze reliably.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, linux, python
Domain
cloud, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.