elastic / elastic/beats

Support for S3 zstd compressed log files in Inputs

Open
#45,292 2 comments 0 reactions 0 assignees View on GitHub
Stalled Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

**Describe the enhancement:**

The documentation states :

[Handling Compressed Objects](https://www.elastic.co/docs/reference/beats/filebeat/filebeat-input-aws-s3#_handling_compressed_objects)

> S3 objects that use the gzip format ([RFC 1952](https://rfc-editor.org/rfc/rfc1952.html)) with the DEFLATE compression algorithm are automatically decompressed during processing. This is achieved by checking for the gzip file magic header.

It would be nice for zstd to be detected (for instance with `logfile.log.zst` files, I guess there is a header that enables this detection too) and uncompressed on the fly.

The zstd file compression is : https://facebook.github.io/zstd/

**Describe a specific use case for the enhancement or feature:**

We've set up lambda that ships it's logs to a log group with DELIVERY log class which in turn ships the logs to S3 (like an ALB would). We realized that the logs where in `.zst` extensions, downloading a file we have to use `unzstd` command to unzip the files.

Contributor guide

Open the contributing guide

Research direction

Start with the Filebeat AWS S3 input and its documented “Handling Compressed Objects” behavior, then read the linked zstd specification to understand detection and decompression. Done means S3 log objects using zstd, including .zst files, are detected and decompressed during processing in addition to the existing gzip behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, go
Domain
backend, cloud
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.