airbytehq / airbytehq/airbyte

S3 Destination: Support partition keys

未关闭
#6,981 12 条评论 28 个 reaction 已指派 1 人 已被 @evantahler 认领 在 GitHub 查看
area/connectors area/platform autoteam connectors/destination/s3 connectors/destinations-files frozen team/destinations type/enhancement
主要语言
Python
星标
22.1k
派生
5.3k
PR 合并指标
PR 指标待抓取

描述

## Tell us about the problem you're trying to solve
To achieve optimal performance, one often needs to partition data into S3 directories based on the value of a field in the incoming records e.g: based on the `date` field, records with date `October` should go into the `october/` directory, those with value `november` should go in `november/` directory etc.

This is pretty important for performance downstream. Often times, S3-based solutions (e.g: redshift, Hive, Trino, etc..) leverage the partition key structure to optimize queries.

This is an interesting problem for us to solve in that it is very connection-specific. Currently no configuration has a way of being customized on a per-connection basis. The partition key for a particular stream is different than for other streams, also between different connections.

## Describe the solution you’d like
I would like to be able to set the partition key for each stream in my connection

贡献指南

打开贡献指南

评估

这个 Issue 还没有评估数据。

把新 issue 发到你的邮箱

精选适合新手参与的 GitHub issue 摘要。