airbytehq / airbytehq/airbyte

S3 Destination: Support partition keys

未關閉
#6,981 12 則留言 28 個 reaction 已指派 1 人 已被 @evantahler 認領 在 GitHub 檢視
area/connectors area/platform autoteam connectors/destination/s3 connectors/destinations-files frozen team/destinations type/enhancement
主要語言
Python
星號
22.1k
分支
5.4k
平均合併
5 小時
30 天內合併 PR
671

描述

## Tell us about the problem you're trying to solve
To achieve optimal performance, one often needs to partition data into S3 directories based on the value of a field in the incoming records e.g: based on the `date` field, records with date `October` should go into the `october/` directory, those with value `november` should go in `november/` directory etc.

This is pretty important for performance downstream. Often times, S3-based solutions (e.g: redshift, Hive, Trino, etc..) leverage the partition key structure to optimize queries.

This is an interesting problem for us to solve in that it is very connection-specific. Currently no configuration has a way of being customized on a per-connection basis. The partition key for a particular stream is different than for other streams, also between different connections.

## Describe the solution you’d like
I would like to be able to set the partition key for each stream in my connection

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。