Azure / Azure/azure-cli

Azure Universal Artifacts download of large package fails

Open
#16,956 7 comments 1 reaction 0 assignees View on GitHub
act-observability-squad Artifacts Service Attention
Dominant language
Python
Stars
4.6k
Forks
3.5k
Avg merge
3d 2h
Merged PRs (30d)
60

Description

**Describe the bug**
Downloading a Universal Package of > 22GB fails when using az cli and self hosted agent.

**To Reproduce**
I uploaded a large universal package of around 22GB to a cloud hosted package feed. This took an appropriate amount of time based on my ISP upload speed. Using the following command then fails:
`az artifacts universal download --organization https://dev.azure.com/XXX --project="Test" --scope project --feed TestFeed --name ansible-media3 --version 2.3.2 --path fred`
It does seem to spend a good 30 minutes downloading what must be the majority of the package before failing with a 403 authorisation error.

**Expected behavior**
I would expect a universal package of any size up to the reported max of 1TB [(here)](https://docs.microsoft.com/en-us/azure/devops/artifacts/reference/limits?view=azure-devops&viewFallbackFrom=vsts) to download given sufficient time and space on the host.

**Environment summary**
Using Windows 10 powershell and:
azure-cli 2.18.0
core 2.18.0
telemetry 1.0.6
Extensions:
azure-devops 0.18.0

**Additional context**
There is a lot of information in an azure devops ticket [here](https://developercommunity2.visualstudio.com/t/azure-artifacts-download-of-large-package-fails/1308520?from=email&viewtype=all#T-ND1340546). A quick summary of what was done. We switched from using `az login` to using a PAT which didn't help. We then moved to self hosted agents which also gave the same error. We tried different accounts within the same organisation and this also didn't help.
In the same environment repeated downloads of 1GB and 4GB universal packages are fine.
My goal is to actually use universal packages of up to 100GB but this 22GB package was my third test of the capability.
We can't adopt Azure Universal Packages if the download fails.

I've attached [downloadexception.txt](https://github.com/Azure/azure-cli/files/5994348/downloadexception.txt) which was the initial error, [downloadwithpat.txt](https://github.com/Azure/azure-cli/files/5994352/downloadwithpat.txt) using a PAT and then [downloadlogwithdebug.txt](https://github.com/Azure/azure-cli/files/5994353/downloadlogwithdebug.txt) which has the debug flag on.

Have people downloaded large universal packages (>22GB) with the CLI?
I'm wondering if there is maybe some timeout option somewhere which means I get 30ish minutes of download before it fails?

Any help would be much appreciated. If this is the wrong place to raise this issue please let me know where to log it as the Azure Devops support after some time suggested here.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.