S3 Destination: Support partition keys
- 主要語言
- Python
- 星號
- 22.1k
- 分支
- 5.4k
- 平均合併
- 5 小時
- 30 天內合併 PR
- 671
描述
## Tell us about the problem you're trying to solve
To achieve optimal performance, one often needs to partition data into S3 directories based on the value of a field in the incoming records e.g: based on the `date` field, records with date `October` should go into the `october/` directory, those with value `november` should go in `november/` directory etc.
This is pretty important for performance downstream. Often times, S3-based solutions (e.g: redshift, Hive, Trino, etc..) leverage the partition key structure to optimize queries.
This is an interesting problem for us to solve in that it is very connection-specific. Currently no configuration has a way of being customized on a per-connection basis. The partition key for a particular stream is different than for other streams, also between different connections.
## Describe the solution you’d like
I would like to be able to set the partition key for each stream in my connection
貢獻指南
評估
這個 Issue 還沒有評估資料。