airbytehq / airbytehq/airbyte

Mapping multiple source streams into a single destination table on Postgres

オープン
#2,224 コメント 6 件 リアクション 3 件 担当者 0 名 GitHub で見る
connectors/destination/postgres connectors/destinations-database frozen lang/java normalization team/destinations transformation type/enhancement
主要言語
Python
スター
22.1k
フォーク
5.3k
PR マージ指標
PR 指標を取得中

説明

## Tell us about the problem you're trying to solve
I have a 3rd party API from which I read industrial sensor data (time-series data) of the form (timestamp, value) for each of the close to 1000 sensors. I'm planning to write a custom HTTP source for this. The source's `discover` method returns one stream per sensor so that I can filter which sensors I want to sync in the Airbyte UI. So, I have close to 1000 source streams. On the Postgres destination (using TimescaleDB), I obviously don't want to create 1000 tables corresponding to 1000 streams. Instead, I want all streams to be merged into a single table with the format (stream_name/sensor name, timestamp, value).

## Describe the solution you’d like
Ability to merge multiple source streams into a single destination table by adding the stream name as an additional column without having to write a custom destination in Airbyte, especially when all the streams have the same schema.

## Describe the alternative you’ve considered or used
Custom pipelines.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368215809205) by [Unito](https://www.unito.io)

コントリビューションガイド

コントリビューションガイドを開く

評価

この issue はまだ評価されていません。

新しい issue をメールで受け取る

初心者向けの GitHub issue を短くまとめたダイジェスト。