facebook / facebook/zstd

Environment variable for --memory

Open
#4,028 3 comments 4 reactions 0 assignees View on GitHub
feature request
Dominant language
C
Stars
27.9k
Forks
2.6k
Avg merge
1d 3h
Merged PRs (30d)
8

Description

**Is your feature request related to a problem? Please describe.**

Sometimes I use tools that utilize zstd, but they don't allow me to pass arbitrary flags to zstd (in this case `rg -z`). In these cases, the tool won't work in cases where the archive requires additional flags to decompress (such as needing `--long=32` or `--memory=2048MB`)

**Describe the solution you'd like**

It would be great if there was an environment variable like ZSTD_MEMORY or similar to specify window size for compression/decompression.

This would allow me to use tools where I'm not able to pass options to zstd to decompress files that would need a flag like `--long=32` or `--memory=2048MB`.

**Describe alternatives you've considered**

In this case, I wrote a shell script using `xargs` that ran `zstdcat --long=31` on files and piped them to `rg`.

**Additional context**

In this case, I was using `rg` to gather statistics on a large compressed text corpora. Usually `rg` is nice for search a lot of compressed files in a directory, but these had been compressed with --long, so I got an error[1].

I also [opened a bug](https://github.com/BurntSushi/ripgrep/issues/2781) in the ripgrep repo to support passing flags to the compression libraries it uses. That said, this would be a really nice general feature of zstd.

Thank you! :)

**Footnotes**

[1] Error message from rg, bubbling up an error from zstd:

```
-------------------------------------------------------------------------------
./submissions/RS_2018-01.zst:
-------------------------------------------------------------------------------
sions/RS_2018-01.zst : Decoding error (36) : Frame requires too much memory for decoding
sions/RS_2018-01.zst : Window size larger than maximum : 2147483648 > 134217728
sions/RS_2018-01.zst : Use --long=31 or --memory=2048MB
-------------------------------------------------------------------------------
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.