elastic / elastic/beats

[REQUEST]: Document AWS S3 Input support for GZ/ZIP files

Open
#40,320 7 comments 0 reactions 0 assignees View on GitHub
Team:Cloud-Monitoring Team:Docs Team:obs-ds-hosted-services
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 2h
Merged PRs (30d)
364

Description

### Description

The AWS S3 Input docs doesn't mention the ability to harvest `.gz/.zip` files in S3 buckets in general.

Seems starting 8.0 + 7.9+ we started supporting decompression of files from AWS even if their extension is not `.gz/.zip` via https://github.com/elastic/beats/pull/18764 via content type detection.

The pages to update are:
- https://www.elastic.co/guide/en/beats/filebeat/current/filebeat-input-aws-s3.html
- Any integration page referencing AWS S3 Input directly (Custom S3 Integration, AWS S3 Integration, ...)

### Resources

I think we should cover the details of this capability.

In particular, we also need to explain/clarify if the use of compressed files might impact the `expand_event_list_from_field` feature (from the code, it seems not running if the content-type is not "application/json").

### Collaboration

Please choose a preferred collaboration model.

### Point of contact.

**Main contact:** @elastic/obs-cloud-monitoring

**Stakeholders:**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.