Azure / Azure/azure-linux-extensions
Azure Backup job is failing to unfreeze Ubuntu OS after running safefreeze
- Dominant language
- Python
- Stars
- 333
- Forks
- 278
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 4
Description
Hello,
this is second time (of 3 times total) when we encountered an error with fsfreeze during Azure Backup. Our system is during the Azure Backup freezed but it's never unfreezed. The whole server is stuck and the only thing we can do is deallocate Azure VM and start it again.
When we checked logs we see that the last thing logged into the extension.log is run of a procedure safefreeze and proceed for accepting singal. Raw log:
2024-01-14 22:16:48.792020 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]PreSnapshot: Status Code: 200
2024-01-14 22:16:48.794220 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]Taking Snapshot through Host
2024-01-14 22:16:48.796446 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]T:S freeze, timeout value 60
2024-01-14 22:16:48.798717 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]skipped mount :
2024-01-14 22:16:48.800908 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]fsfreeze mount :/mnt
2024-01-14 22:16:48.803464 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]fsfreeze mount :/
2024-01-14 22:16:48.805675 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]skip freeze is : False
2024-01-14 22:16:48.811102 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]arg : ['/var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9207.0/main/safefreeze/bin/safefreeze', '60', '/']
2024-01-14 22:16:48.813571 [Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0]proceeded for accepting signals
In Azure Backup job report we see the job as failed with these details:
ExtensionOperationInProgress
Command execution failed.
Another operation is in progress on this item. Please wait until the previous operation is completed.
We set everything correct like this documentation states:
https://learn.microsoft.com/en-us/azure/backup/backup-azure-linux-app-consistent
We are unable to identify the core of the issue deeper. We assume that it's a bug because the system is never unfreezed as it should be. We can't affect this behavior because it's initialized from Azure Backup agent job.
Our OS:
Ubuntu 18.04.6 LTS
VM agent version:
2.7.3.0
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the raw extension.log entries for the safefreeze invocation at /var/lib/waagent/Microsoft.Azure.RecoveryServices.VMSnapshotLinux-1.0.9207.0/main/safefreeze/bin/safefreeze and review the Azure Linux app-consistent backup documentation. Reproduce the Azure Backup operation on Ubuntu 18.04.6 and trace the freeze and signal-handling steps; done means the backup completes without leaving the OS frozen.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python, ubuntu
- Domain
- cloud, devops, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100