microsoft / microsoft/AzureStorageExplorer
decompression worker exited early. When Flushing file
Nobody has claimed this yet.
- Dominant language
- No language data
- Stars
- 452
- Forks
- 92
- Avg merge
- 15h 20m
- Merged PRs (30d)
- 3
Description
I am downloading files in bulk from a single azure blob storage container using Azure Storage Explorer.
I have connected to the container using a SAS URL.
However, for some files (100 out of 4000) I see the following error in the logs:
2022/07/04 11:49:28 ERR: [P#0-T#4131] DOWNLOADFAILED: <file_URL_redacted>: decompression worker exited early. When Flushing file. X-Ms-Request-Id:
I am able to download the files directly from the Azure Portal. Lots of the files have CONTENT-ENCODING deflate and the Setting in Azure Storage Explorer to Decompress Files is enabled.
This seems to stop the bulk download after ~4000-5000 files. If I start again from where it stopped, I can download another 4000-5000 files before the issue occurs again?
I also see errors like "Error closing file: timed out closing decompression worker"
Is this because too many downloads are happening concurrently?
Any recommendations please?
Head of log file produced from Azure Storage Explorer:
2022/07/04 10:34:22 AzcopyVersion 10.14.1
2022/07/04 10:34:22 OS-Environment windows
2022/07/04 10:34:22 OS-Architecture amd64
...
2022/07/04 10:34:22 Number of CPUs: 4
2022/07/04 10:34:22 Max file buffer RAM 2.000 GB
2022/07/04 10:34:22 Max file buffer RAM 2.000 GB
2022/07/04 10:34:22 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2022/07/04 10:34:22 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2022/07/04 10:34:22 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2022/07/04 10:34:22 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2022/07/04 10:34:22 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2022/07/04 10:34:22 Max open files when downloading: 2147483224 (auto-computed)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reproducing the bulk download in Azure Storage Explorer with a SAS URL, decompression enabled, and the reported concurrency settings. Compare the decompression-worker and file-closing errors while varying AZCOPY_CONCURRENCY_VALUE and AZCOPY_CONCURRENT_FILES; done means identifying the failure cause and establishing a reliable fix or configuration recommendation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100