apache / apache/iggy

feat(connectors): add compression support to S3 sink connector (gzip, zstd)

Open
#3,104 2 comments 0 reactions 0 assignees View on GitHub
connectors enhancement rust
Dominant language
Rust
Stars
4.9k
Forks
432
Avg merge
2d 10h
Merged PRs (30d)
173

Description

## Description

The S3 sink connector (`iggy_connector_s3_sink`) added in #2976 currently writes uncompressed files to S3. For production workloads with high message throughput, compression is essential to reduce storage costs and upload times.

## Proposed Changes

Add a `compression` config option to the S3 sink connector supporting:

- **`none`** (default, current behavior)
- **`gzip`** — widely supported, good compatibility with downstream tools (Athena, Spark, etc.)
- **`zstd`** — better compression ratio and speed, growing ecosystem support

### Config Example

```toml
[plugin_config]
compression = "gzip" # none | gzip | zstd
```

#### Implementation Notes

- Compress the finalized buffer bytes before uploading to S3 (after finalize_buffer, before upload_with_retry)
- Append the appropriate file extension (.jsonl.gz, .jsonl.zst, .json.gz, etc.) — the path module already derives extensions from OutputFormat, this needs to account for compression
- Set the correct Content-Encoding header on the S3 put_object call
- Use flate2 for gzip and zstd crate for zstd — check if they're already in the workspace, otherwise add them
- Add unit tests for round-trip compress/decompress verification
- Update the connector README and config.toml example

Contributor guide

Open the contributing guide

Research direction

Start in the iggy_connector_s3_sink implementation, tracing finalize_buffer through upload_with_retry and the path module to understand buffer handling, filenames, and S3 uploads. Add the compression option, extensions, headers, round-trip unit tests, and the README and config.toml example; done means none, gzip, and zstd uploads work with the documented configuration.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
cloud, data-engineering
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.