facebook / facebook/zstd

Potential Issue: Hang When Used with tar --use-compress-program on Large Files (>300 MB) in Windows PowerShell

Open
#4,432 4 comments 1 reaction 0 assignees View on GitHub
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

**Description:**
Hello zstd team!

I would like to report a potential issue when using `zstd.exe` as a compression program in a pipeline with the `tar` command on Windows, specifically in PowerShell. I am experiencing an indefinite hang when attempting to compress directories or large files using the command `tar --use-compress-program 'zstd.exe -T0 -19' -cf`. The issue occurs consistently when the total directory size exceeds approximately 350 MB or when a single file is larger than 300 MB.

**Steps to Reproduce:**
1. Use the native `tar.exe` from Windows or the version from Git for Windows, along with `zstd.exe` (version 1.5.7) on Windows 10/11 or Windows Server.
2. Create a test directory (e.g., `C:\backup`) with a total size exceeding 350 MB or a single file larger than 300 MB.
3. Run the following command in PowerShell:
```
tar --use-compress-program "zstd.exe -T0 -19" -cf C:\backup\backup.tar.zst -C C:\backup Backup_Folder
```
4. Observe that the process hangs indefinitely (no explicit error, it just remains running).

**Expected Behavior:**
The command should complete the compression of the directory or large file, generating the `backup.tar.zst` file without hanging.

**Observed Behavior:**
The process hangs, consuming resources (CPU usage drops to 0% while RAM usage typically remains above 900 MB) with no visible progress. Interrupting with Ctrl+C is required. The same command works for smaller directories (<350 MB) or files (<300 MB).

**Additional Tests:**
- Using two separate commands (`tar -cf` followed by `zstd -T0 -19`) resolves the issue, suggesting the hang may be related to the pipeline.
- Reducing the compression level (e.g., `-T1 -6`) does not prevent the hang.
- Processing in batches avoids the hang but is not an ideal solution for large volumes.
- Tested with 7-Zip to create `.tar` and then compress with `zstd.exe`, which works, but I prefer to use `tar` for compatibility and to reduce the number of commands in scripts.
- On Linux, using the command `tar -I 'zstd -T0 -19' -cf` with directories and files exceeding 1 GB, including a single file, did not result in any issues.

**Environment:**
- Operating System: Windows 10/11 or Windows Server 2022
- PowerShell: 5.1 or 7.x
- tar.exe: bsdtar 3.7.7 - libarchive 3.7.7 zlib/1.2.13.1-motley liblzma/5.4.3 bz2lib/1.0.8 libzstd/1.5.5
- zstd.exe: 1.5.7 (downloaded from [https://github.com/facebook/zstd/releases](https://github.com/facebook/zstd/releases))
- Date: July 10, 2025

**Question:**
Could this be a known limitation of `zstd` when handling large data streams in pipelines on Windows? Or might it be a compatibility issue with the version of `tar` being used? I would appreciate any guidance or suggestions to work around this or confirm if it’s a bug worth investigating.

Thank you for your attention and for the excellent work on zstd!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.