Azure ML workspace stuck in soft-deleted state: create blocked, permanently-delete cannot find workspace
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
### Describe the bug
Azure CLI Version:
2.88.0
ML Extension:
2.44.1
Subscription:
Resource Group:
stmlwus2ws01-rg
Workspace:
stmlwus2ws01
Problem:
I deleted an Azure ML workspace.
When I attempt to recreate the workspace, deployment fails with:
"Soft-deleted workspace exists. Please purge or recover it."
However, attempting to permanently delete the workspace fails because the workspace cannot be found:
az ml workspace delete \
--name myws \
--resource-group my-rg \
--subscription \
--permanently-delete \
--yes
Error:
(UserError) Unable to find workspace:
/subscriptions/.../resourcegroups/my-rg/providers/Microsoft.MachineLearningServices/workspaces/myws
This creates a deadlock:
- Workspace creation fails because a soft-deleted workspace exists.
- Permanent delete fails because the workspace cannot be found.
- No CLI command appears available to enumerate or purge the soft-deleted workspace.
Expected behavior:
Either:
1. `az ml workspace delete --permanently-delete` should purge the soft-deleted workspace, or
2. The CLI should provide a supported command to list and purge soft-deleted workspaces.
Actual behavior:
Workspace name remains blocked but cannot be purged.
### Related command
az ml workspace delete \
--name myws \
--resource-group my-rg \
--subscription \
--permanently-delete \
--yes
### Errors
(UserError) Unable to find workspace:
Code: UserError
Message: Unable to find workspace:
### Issue script & Debug output
> cli: Received HttpResponseError: Traceback (most recent call last):
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
cli.azure.cli.core.azclierror: Traceback (most recent call last):
File "/opt/az/lib/python3.14/site-packages/knack/cli.py", line 233, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/__init__.py", line 677, in execute
raise ex
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/__init__.py", line 820, in _run_jobs_serially
results.append(self._run_job(expanded_arg, cmd_copy))
~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/__init__.py", line 789, in _run_job
result = cmd_copy(params)
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/__init__.py", line 335, in __call__
return self.handler(*args, **kwargs)
~~~~~~~~~~~~^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/cli/core/commands/command_operation.py", line 120, in handler
return op(**command_args)
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 285, in ml_workspace_delete
log_and_raise_error(err, debug)
~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/raise_error.py", line 193, in log_and_raise_error
raise cli_error
File "/home/myubuntuuser/.azure/cliextensions/ml/azext_mlv2/manual/custom/workspace.py", line 278, in ml_workspace_delete
del_result = ml_client.workspaces.begin_delete(
name=name, delete_dependent_resources=all_resources, permanently_delete=permanently_delete
)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 355, in begin_delete
return super().begin_delete(
~~~~~~~~~~~~~~~~~~~~^
name, delete_dependent_resources=delete_dependent_resources, permanently_delete=permanently_delete, **kwargs
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
)
^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 494, in begin_delete
workspace: Any = self.get(name, **kwargs)
~~~~~~~~^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_telemetry/activity.py", line 288, in wrapper
return f(*args, **kwargs)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations.py", line 143, in get
return super().get(workspace_name=name, **kwargs)
~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/operations/_workspace_operations_base.py", line 91, in get
obj = self._operation.get(resource_group, workspace_name)
File "/opt/az/lib/python3.14/site-packages/azure/core/tracing/decorator.py", line 119, in wrapper_use_tracer
return func(*args, **kwargs)
File "/home/myubuntuuser/.azure/cliextensions/ml/azure/ai/ml/_restclient/v2024_10_01_preview_tsp/operations/_operations.py", line 10422, in get
map_error(status_code=response.status_code, response=response, error_map=error_map)
~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/az/lib/python3.14/site-packages/azure/core/exceptions.py", line 163, in map_error
raise error
azure.core.exceptions.ResourceNotFoundError: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
cli.azure.cli.core.azclierror: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
az_command_data_logger: (ResourceNotFound) The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
Code: ResourceNotFound
Message: The Resource 'Microsoft.MachineLearningServices/workspaces/mymlws' under resource group 'mymlws-rg' was not found. For more details please go to https://aka.ms/ARMResourceNotFoundFix
cli.knack.cli: Event: Cli.PostExecute []
az_command_data_logger: exit code: 3
cli.__main__: Command ran in 1.545 seconds (init: 0.049, invoke: 1.497)
telemetry.main: Begin splitting cli events and extra events, total events: 1
telemetry.main: Finish splitting cli events and extra events, cli events: 1
telemetry.save: Save telemetry record of length 4607 in cache file under /home/myubuntuuser/.azure/telemetry/20260720191821574
telemetry.main: Begin creating telemetry upload process.
telemetry.process: Creating upload process: "/opt/az/bin/python3 /opt/az/lib/python3.14/site-packages/azure/cli/telemetry/__init__.py /home/myubuntuuser/.azure /home/myubuntuuser/.azure/telemetry/20260720191821574"
telemetry.process: Return from creating process 1362084
telemetry.main: Finish creating telemetry upload process.
### Expected behavior
purged the soft-deleted workspacee
### Environment Summary
{
"azure-cli": "2.88.0",
"azure-cli-core": "2.88.0",
"azure-cli-telemetry": "1.1.0",
"extensions": {
"ml": "2.44.1"
}
}
which python3
/usr/bin/python3
Python 3.14.4
### Additional context
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.