Potential Issue: Hang When Used with tar --use-compress-program on Large Files (>300 MB) in Windows PowerShell
- Dominant language
- C
- Stars
- 27.9k
- Forks
- 2.6k
- Avg merge
- 1d 3h
- Merged PRs (30d)
- 8
Description
**Description:**
Hello zstd team!
I would like to report a potential issue when using `zstd.exe` as a compression program in a pipeline with the `tar` command on Windows, specifically in PowerShell. I am experiencing an indefinite hang when attempting to compress directories or large files using the command `tar --use-compress-program 'zstd.exe -T0 -19' -cf`. The issue occurs consistently when the total directory size exceeds approximately 350 MB or when a single file is larger than 300 MB.
**Steps to Reproduce:**
1. Use the native `tar.exe` from Windows or the version from Git for Windows, along with `zstd.exe` (version 1.5.7) on Windows 10/11 or Windows Server.
2. Create a test directory (e.g., `C:\backup`) with a total size exceeding 350 MB or a single file larger than 300 MB.
3. Run the following command in PowerShell:
```
tar --use-compress-program "zstd.exe -T0 -19" -cf C:\backup\backup.tar.zst -C C:\backup Backup_Folder
```
4. Observe that the process hangs indefinitely (no explicit error, it just remains running).
**Expected Behavior:**
The command should complete the compression of the directory or large file, generating the `backup.tar.zst` file without hanging.
**Observed Behavior:**
The process hangs, consuming resources (CPU usage drops to 0% while RAM usage typically remains above 900 MB) with no visible progress. Interrupting with Ctrl+C is required. The same command works for smaller directories (<350 MB) or files (<300 MB).
**Additional Tests:**
- Using two separate commands (`tar -cf` followed by `zstd -T0 -19`) resolves the issue, suggesting the hang may be related to the pipeline.
- Reducing the compression level (e.g., `-T1 -6`) does not prevent the hang.
- Processing in batches avoids the hang but is not an ideal solution for large volumes.
- Tested with 7-Zip to create `.tar` and then compress with `zstd.exe`, which works, but I prefer to use `tar` for compatibility and to reduce the number of commands in scripts.
- On Linux, using the command `tar -I 'zstd -T0 -19' -cf` with directories and files exceeding 1 GB, including a single file, did not result in any issues.
**Environment:**
- Operating System: Windows 10/11 or Windows Server 2022
- PowerShell: 5.1 or 7.x
- tar.exe: bsdtar 3.7.7 - libarchive 3.7.7 zlib/1.2.13.1-motley liblzma/5.4.3 bz2lib/1.0.8 libzstd/1.5.5
- zstd.exe: 1.5.7 (downloaded from [https://github.com/facebook/zstd/releases](https://github.com/facebook/zstd/releases))
- Date: July 10, 2025
**Question:**
Could this be a known limitation of `zstd` when handling large data streams in pipelines on Windows? Or might it be a compatibility issue with the version of `tar` being used? I would appreciate any guidance or suggestions to work around this or confirm if it’s a bug worth investigating.
Thank you for your attention and for the excellent work on zstd!
Contributor guide
Assessment
This issue has not been assessed yet.