Perform blob download-batch concurrently
- Dominant language
- Python
- Stars
- 4.6k
- Forks
- 3.5k
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 60
Description
**Related command**
az storage blob download-batch
**Is your feature request related to a problem? Please describe.**
I have a need to download a large number of small blobs. I set the "--max-connections" property thinking this would allow for multiple files to download at once, but this does not appear to be the case. I'm downloading these blobs to an Azure VM (with more than 20 cores) in the same region and network receive rate is hovering around 1 MiB/second. I'd expect that rate to feasibly be well over 100 MiB/second or more with concurrency.
**Describe the solution you'd like**
My understanding reading through the code repository is that this functionality is handled with this function:
https://github.com/Azure/azure-cli/blob/a55543015da9e2f554a6a09816794b5315e3ce8b/src/azure-cli/azure/cli/command_modules/storage/operations/blob.py#L426
At a minimum, I'd like to see the for loop which loops over the `blobs_to_download` dictionary be converted into an asynchronous process. This either means that `_download_blob` function (and everything lower in the stack) would need to be async functions. Or an async run in thread.
It would be even more awesome if the `collect_blobs` function could return an async generator, which could allow the downloads to start before the full list of blobs has been collected (though I realize this has more side effects). My current download took several hours to accumulate the list of blobs to download (since the API list call is paginated).
**Describe alternatives you've considered**
There isn't really any great alternatives other than rewriting my own version of this api which is undesirable.
**Additional context**
The sample download-batch I am running is 12,024,850 blobs with an average size of 10KiB. The download is running on an Azure VM of size: NC24ads_A100_v4 in the same region (East US 2) as the storage account. My expectation is that this would feasibly be able to be complete in under 2 hours, yet we are at about 14 hours and not yet to 30% complete yet.
Contributor guide
Assessment
This issue has not been assessed yet.