S3 Destination: Support partition keys
- 主要语言
- Python
- 星标
- 22.1k
- 派生
- 5.3k
- PR 合并指标
- PR 指标待抓取
描述
## Tell us about the problem you're trying to solve
To achieve optimal performance, one often needs to partition data into S3 directories based on the value of a field in the incoming records e.g: based on the `date` field, records with date `October` should go into the `october/` directory, those with value `november` should go in `november/` directory etc.
This is pretty important for performance downstream. Often times, S3-based solutions (e.g: redshift, Hive, Trino, etc..) leverage the partition key structure to optimize queries.
This is an interesting problem for us to solve in that it is very connection-specific. Currently no configuration has a way of being customized on a per-connection basis. The partition key for a particular stream is different than for other streams, also between different connections.
## Describe the solution you’d like
I would like to be able to set the partition key for each stream in my connection
贡献指南
评估
这个 Issue 还没有评估数据。