feat(connectors): add compression support to S3 sink connector (gzip, zstd)
- Dominant language
- Rust
- Stars
- 4.9k
- Forks
- 432
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 173
Description
## Description
The S3 sink connector (`iggy_connector_s3_sink`) added in #2976 currently writes uncompressed files to S3. For production workloads with high message throughput, compression is essential to reduce storage costs and upload times.
## Proposed Changes
Add a `compression` config option to the S3 sink connector supporting:
- **`none`** (default, current behavior)
- **`gzip`** — widely supported, good compatibility with downstream tools (Athena, Spark, etc.)
- **`zstd`** — better compression ratio and speed, growing ecosystem support
### Config Example
```toml
[plugin_config]
compression = "gzip" # none | gzip | zstd
```
#### Implementation Notes
- Compress the finalized buffer bytes before uploading to S3 (after finalize_buffer, before upload_with_retry)
- Append the appropriate file extension (.jsonl.gz, .jsonl.zst, .json.gz, etc.) — the path module already derives extensions from OutputFormat, this needs to account for compression
- Set the correct Content-Encoding header on the S3 put_object call
- Use flate2 for gzip and zstd crate for zstd — check if they're already in the workspace, otherwise add them
- Add unit tests for round-trip compress/decompress verification
- Update the connector README and config.toml example
Contributor guide
Research direction
Start in the iggy_connector_s3_sink implementation, tracing finalize_buffer through upload_with_retry and the path module to understand buffer handling, filenames, and S3 uploads. Add the compression option, extensions, headers, round-trip unit tests, and the README and config.toml example; done means none, gzip, and zstd uploads work with the documented configuration.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- cloud, data-engineering
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100