Base workspace delete fails with DeleteShareWhenSnapshotLeased when enable_backup=true and delete_backups_on_uninstall=false (defaults)

Summary

Deleting a base workspace that has backups enabled but is uninstalled with delete_backups_on_uninstall=false (both are the shipped defaults) fails and leaves the workspace stuck in deleting_failed. Terraform aborts with:

Error: Failed to delete resource
deleting Resource ... storageAccounts/stgws<id>/fileServices/default/shares/vm-shared-storage
RESPONSE 409: DeleteShareWhenSnapshotLeased

Azure Files — the lease is on the share snapshot and is what makes deletion non-trivial

When Azure Backup protects an Azure Files share, each recovery point is a snapshot of the share, and Backup places an infinite-duration lease on that snapshot to guard against accidental deletion. Per the docs:

• About Azure Files backup — How lease snapshot works: "Azure Backup … adds a lock on the snapshot using the lease snapshot capability … the lock's duration is infinite. If a File Share has leased-snapshots, the deletion is no more a one-click operation."
• Azure Files backup FAQ — Lease snapshots: the recommended way to delete a backed-up share is "stop protection with delete data … Azure Backup will release the lease and delete all snapshots. Then you can delete the file share."

Crucially, stopping protection does not release the lease — the lease is infinite-duration and the retained recovery point does not age out once protection is stopped-with-retain, so only stop protection with delete data removes the recovery points and releases it. (This asymmetry is Files-specific: VM backup does not block VM deletion — Manage recovery points — Impact of deleting a VM.)

Root cause

In templates/workspaces/base/remove_backup.sh the retain branch:

else
  remove_from_state "${backup_resources[@]}" "azurerm_resource_group.ws"
  ...
  echo "Backup resources retained. Terraform will delete the other workspace resources and leave the vault and its resource group in place."
fi

• Runs stop_protection with --delete-backup-data false → recovery point retained, snapshot lease still held (stopping alone doesn't release it).
• Removes only the backup objects and azurerm_resource_group.ws from state — but leaves azurerm_storage_account / azurerm_storage_share in state.
• Terraform therefore still tries to destroy vm-shared-storage, which is blocked by the retained, leased snapshot → 409.

So the "retain" path neither cleanly retains (it tries to delete the share) nor succeeds (the lease blocks it). Every resource-processor retry repeats identically.

Evidence

On a stuck workspace (protection already stopped):

protectionState: ProtectionStopped
recovery points:  1        # snapshot + lease survive the stop

Impact

Defaults are enable_backup=true and delete_backups_on_uninstall=false, so a default delete of a backed-up base workspace gets stuck in deleting_failed and needs manual backup-data deletion to unblock.

Proposed fix (pick one)

1. In the retain branch, also drop azurerm_storage_account / azurerm_storage_share (and other RG-scoped resources) from state so Terraform truly leaves the whole resource group intact — matching the log message's intent.
2. Or, always release the snapshot leases before the Terraform teardown by doing stop protection with delete data for the file-share item (even when retaining the vault elsewhere), so the share can be destroyed.
3. At minimum, document that deleting a backed-up workspace requires delete_backups_on_uninstall=true, and surface a clearer error.

Note: Terraform cannot resolve this declaratively — the snapshots are Azure Backup-owned (not in state) and azurerm_storage_share has no "include leased snapshots" delete option, so the lease release must remain an Azure Backup operation (script/provisioner), not a provider capability.

Workaround

Run the true-path effect manually per workspace: az backup protection disable ... --delete-backup-data true --yes (+ delete the vault), then re-trigger the workspace delete; or set delete_backups_on_uninstall=true before deleting.

Start in templates/workspaces/base/remove_backup.sh and compare the retain branch with the true delete-backups path, then inspect how Terraform state handles the storage account and share. Verify the Azure Backup lease and recovery-point behavior for a retained file-share backup. Done means deleting a default backed-up workspace no longer leaves it in deleting_failed, while the selected retention behavior matches the log message.