microsoft / microsoft/AzureStorageExplorer

decompression worker exited early. When Flushing file

Open
#5,899 14 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

:gear: azcopy
Dominant language
No language data
Stars
452
Forks
92
Avg merge
15h 20m
Merged PRs (30d)
3

Description

I am downloading files in bulk from a single azure blob storage container using Azure Storage Explorer.

I have connected to the container using a SAS URL.

However, for some files (100 out of 4000) I see the following error in the logs:

2022/07/04 11:49:28 ERR: [P#0-T#4131] DOWNLOADFAILED: <file_URL_redacted>: decompression worker exited early. When Flushing file. X-Ms-Request-Id:

I am able to download the files directly from the Azure Portal. Lots of the files have CONTENT-ENCODING deflate and the Setting in Azure Storage Explorer to Decompress Files is enabled.

This seems to stop the bulk download after ~4000-5000 files. If I start again from where it stopped, I can download another 4000-5000 files before the issue occurs again?

I also see errors like "Error closing file: timed out closing decompression worker"
Is this because too many downloads are happening concurrently?

Any recommendations please?


Head of log file produced from Azure Storage Explorer:

2022/07/04 10:34:22 AzcopyVersion 10.14.1
2022/07/04 10:34:22 OS-Environment windows
2022/07/04 10:34:22 OS-Architecture amd64
...
2022/07/04 10:34:22 Number of CPUs: 4
2022/07/04 10:34:22 Max file buffer RAM 2.000 GB
2022/07/04 10:34:22 Max file buffer RAM 2.000 GB
2022/07/04 10:34:22 Max concurrent network operations: 32 (Based on number of CPUs. Set AZCOPY_CONCURRENCY_VALUE environment variable to override)
2022/07/04 10:34:22 Check CPU usage when dynamically tuning concurrency: true (Based on hard-coded default. Set AZCOPY_TUNE_TO_CPU environment variable to true or false override)
2022/07/04 10:34:22 Max concurrent transfer initiation routines: 64 (Based on hard-coded default. Set AZCOPY_CONCURRENT_FILES environment variable to override)
2022/07/04 10:34:22 Max enumeration routines: 16 (Based on hard-coded default. Set AZCOPY_CONCURRENT_SCAN environment variable to override)
2022/07/04 10:34:22 Parallelize getting file properties (file.Stat): false (Based on AZCOPY_PARALLEL_STAT_FILES environment variable)
2022/07/04 10:34:22 Max open files when downloading: 2147483224 (auto-computed)

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the bulk download in Azure Storage Explorer with a SAS URL, decompression enabled, and the reported concurrency settings. Compare the decompression-worker and file-closing errors while varying AZCOPY_CONCURRENCY_VALUE and AZCOPY_CONCURRENT_FILES; done means identifying the failure cause and establishing a reliable fix or configuration recommendation.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure
Domain
cloud
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.