Support for S3 zstd compressed log files in Inputs
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
**Describe the enhancement:**
The documentation states :
[Handling Compressed Objects](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-aws-s3#_handling_compressed_objects)
> S3 objects that use the gzip format ([RFC 1952](https://rfc-editor.org/rfc/rfc1952.html)) with the DEFLATE compression algorithm are automatically decompressed during processing. This is achieved by checking for the gzip file magic header.
It would be nice for zstd to be detected (for instance with `logfile.log.zst` files, I guess there is a header that enables this detection too) and uncompressed on the fly.
The zstd file compression is : https://facebook.github.io/zstd/
**Describe a specific use case for the enhancement or feature:**
We've set up lambda that ships it's logs to a log group with DELIVERY log class which in turn ships the logs to S3 (like an ALB would). We realized that the logs where in `.zst` extensions, downloading a file we have to use `unzstd` command to unzip the files.
Contributor guide
Research direction
Start with the Filebeat AWS S3 input and its documented “Handling Compressed Objects” behavior, then read the linked zstd specification to understand detection and decompression. Done means S3 log objects using zstd, including .zst files, are detected and decompressed during processing in addition to the existing gzip behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- backend, cloud
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100