Azure / Azure/azure-cli

Getting "The process cannot access the file because it is being used by another process" errors with `~\.azure\logs\az.log`

Open
#30,145 2 comments 0 reactions 1 assignee Claimed by @zhoxing-ms View on GitHub
act-identity-squad ARM Auto-Assign customer-reported question Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

### Describe the bug

When I call `az-cli` commands from PowerShell Core inside parallel jobs, they sometimes run into python's `PermissionError: [WinError 32]` when accessing the log file.

### Related command

I register some resource providers in a new RG with the following `-Parallel` loop:

```powershell
$resourceProviders| ForEach-Object -Parallel {
$SubscriptionId = $using:SubscriptionId
az provider register --wait --subscription $SubscriptionId --namespace $_
} -ThrottleLimit 10
```

This is a similar loop using `ThreadJob`:

```powershell
$jobs = @()
foreach ($rp in $resourceProviders) {
$jobs += Start-ThreadJob -Name $rp -ScriptBlock {
$params = $using:SubscriptionId
az provider register --wait --subscription $SubscriptionId --namespace $rp
}
}
Wait-Job -Job $jobs
```

### Errors

```text
--- Logging error ---
Traceback (most recent call last):
File "logging\handlers.py", line 74, in emit
File "logging\handlers.py", line 179, in doRollover
File "logging\handlers.py", line 115, in rotate
PermissionError: [WinError 32] The process cannot access the file because it is being used by another process: 'C:\\Users\\danmacode\\.azure\\logs\\az.log' -> 'C:\\Users\\danmacode\\.azure\\logs\\az.log.1'
Call stack:
File "", line 198, in _run_module_as_main
File "", line 88, in _run_code
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 41, in
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/__main__.py", line 27, in cli_main
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\knack/cli.py", line 233, in invoke
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 659, in execute
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 733, in _run_jobs_serially
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 703, in _run_job
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/__init__.py", line 336, in __call__
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/commands/command_operation.py", line 361, in handler
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/tracing/decorator.py", line 76, in wrapper_use_tracer
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/mgmt/resource/resources/v2022_09_01/operations/_operations.py", line 8338, in get
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 213, in run
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 70, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 70, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 70, in send
[Previous line repeated 2 more times]
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/mgmt/core/policies/_base.py", line 47, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/policies/_redirect.py", line 181, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/policies/_retry.py", line 467, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/policies/_authentication.py", line 115, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 70, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_base.py", line 75, in send
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/core/pipeline/_tools.py", line 45, in await_result
File "D:\a\_work\1\s\build_scripts\windows\artifacts\cli\Lib\site-packages\azure/cli/core/sdk/policies.py", line 81, in on_response
File "logging\__init__.py", line 1477, in debug
File "logging\__init__.py", line 1634, in _log
File "logging\__init__.py", line 1644, in handle
File "logging\__init__.py", line 1706, in callHandlers
File "logging\__init__.py", line 978, in handle
File "logging\handlers.py", line 77, in emit
Message: 'Response content:'
Arguments: ()
```

### Issue script & Debug output

No debug output.

### Expected behavior

There shouldn't be a permission error when multiple processes or threads try to write to the `az.log` file simultaneously.

### Environment Summary

```text
azure-cli 2.65.0

core 2.65.0
telemetry 1.1.0

Extensions:
account 0.2.5
automation 1.0.0b1
init 0.1.0
next 0.1.3

Dependencies:
msal 1.31.0
azure-mgmt-resource 23.1.1

Python location 'C:\Program Files\Microsoft SDKs\Azure\CLI2\python.exe'
Extensions directory 'C:\Users\danmacode\.azure\cliextensions'

Python (Windows) 3.11.8 (tags/v3.11.8:db85d51, Feb 6 2024, 22:03:32) [MSC v.1937 64 bit (AMD64)]

Legal docs and information: aka.ms/AzureCliLegal

Your CLI is up-to-date.
```

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.