Azure / Azure/azure-devops-cli-extension

Race condition in get_cache_dir

Open
#1,270 0 comments 5 reactions 0 assignees View on GitHub
Feature
Dominant language
Python
Stars
682
Forks
278
Avg merge
3d 23h
Merged PRs (30d)
3

Description

There is a race condition in the `get_cache_dir` function in *_file_cache.py*. If two instances of the CLI are launched at nearly the same time, they will race to create the SDK cache directory. One will win, and the other will throw a `FileExistsError`.

# Proposed Fix

Add `exist_ok=True` to the `os.makedirs` call in *_file_cache.py*:

https://github.com/Azure/azure-devops-cli-extension/blob/0267b759fc9a1a584015973193affdfd8ea4969b/azure-devops/azext_devops/devops_sdk/_file_cache.py#L117

# Workaround

As a workaround, I can run any single `az devops` command to create the cache dir. Then, the race condition is no longer triggered because the cache dir already exists.

# Repro Steps

This is difficult to repro on my local workstation because of thread scheduler variability, but it seems to repro pretty reliably in a CI pipeline. Here's a minimal ADO pipeline to illustrate the issue:

```yaml
trigger: none
pr: none

steps:
- pwsh: |
# Uncomment to make this issue go away (issue does not repro if the cache dir already exists)
#az devops project show -p my-proj-name

$commonArgs = @(
"artifacts", "universal", "download"
"--organization", "https://microsoft.visualstudio.com/"
"--project", "my-proj-name"
"--scope", "project"
"--feed", "my-feed-name"
)

$pkg1Args = $commonArgs + @(
"--name", "test-upack"
"--version", "1.0.0"
"--path", '$(Build.ArtifactStagingDirectory)/test-upack'
)

$pkg2Args = $commonArgs + @(
"--name", "test-upack-two"
"--version", "1.0.0"
"--path", '$(Build.ArtifactStagingDirectory)/test-upack-two'
)

# Start both downloads in parallel
$proc1 = Start-Process -NoNewWindow -FilePath az -ArgumentList $pkg1Args -PassThru
$proc2 = Start-Process -NoNewWindow -FilePath az -ArgumentList $pkg2Args -PassThru

Wait-Process $proc1.Id,$proc2.Id
env:
AZURE_DEVOPS_EXT_PAT: $(System.AccessToken)
```

And here is the output from the above pipeline. As you can see, one of the command instances throws a `FileExistsError` while the other completes successfully and emits a JSON object with info about the downloaded package.

```
ERROR: The command failed with an unexpected error. Here is the traceback:
ERROR: [Errno 17] File exists: '/home/vsts/.azure-devops/python-sdk/cache'
Traceback (most recent call last):
File "/opt/az/lib/python3.8/site-packages/knack/cli.py", line 231, in invoke
cmd_result = self.invocation.execute(args)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 556, in execute
self.commands_loader.load_arguments(command)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/__init__.py", line 515, in load_arguments
command_table[command].load_arguments() # this loads the arguments via reflection
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/__init__.py", line 313, in load_arguments
super(AzCliCommand, self).load_arguments()
File "/opt/az/lib/python3.8/site-packages/knack/commands.py", line 104, in load_arguments
cmd_args = self.arguments_loader()
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 125, in arguments_loader
op = self.get_op_handler(self.op_path)
File "/opt/az/lib/python3.8/site-packages/azure/cli/core/commands/command_operation.py", line 59, in get_op_handler
handler = import_module(mod_to_import)
File "/opt/az/lib/python3.8/importlib/__init__.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 843, in exec_module
File "", line 219, in _call_with_frames_removed
File "/opt/az/azcliextensions/azure-devops/azext_devops/dev/artifacts/universal.py", line 9, in
from azext_devops.dev.common.services import resolve_instance, resolve_instance_and_project
File "/opt/az/azcliextensions/azure-devops/azext_devops/dev/common/services.py", line 13, in
from azext_devops.devops_sdk.connection import Connection
File "/opt/az/azcliextensions/azure-devops/azext_devops/devops_sdk/connection.py", line 5, in
from ._file_cache import RESOURCE_CACHE as RESOURCE_FILE_CACHE
File "/opt/az/azcliextensions/azure-devops/azext_devops/devops_sdk/_file_cache.py", line 113, in
DEFAULT_CACHE_DIR = get_cache_dir()
File "/opt/az/azcliextensions/azure-devops/azext_devops/devops_sdk/_file_cache.py", line 108, in get_cache_dir
os.makedirs(azure_devops_cache_dir)
File "/usr/bin/../../opt/az/lib/python3.8/os.py", line 223, in makedirs
mkdir(name, mode)
FileExistsError: [Errno 17] File exists: '/home/vsts/.azure-devops/python-sdk/cache'
To open an issue, please run: 'az feedback'
{
"ManifestId": "0B8F236189D4B34F6180806225788D5CE118EFB0928ED6E0E2712E3D352DF7DE01",
"PackageSize": 184,
"SuperRootId": "2BB1E6CFD3F78B745AC71B94BA290B594E5AEBBA8B648EA0CB267EB1D5AC8C3302",
"Version": "1.0.0"
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.