Mapping multiple source streams into a single destination table on Postgres
- 主要言語
- Python
- スター
- 22.1k
- フォーク
- 5.3k
- PR マージ指標
- PR 指標を取得中
説明
## Tell us about the problem you're trying to solve
I have a 3rd party API from which I read industrial sensor data (time-series data) of the form (timestamp, value) for each of the close to 1000 sensors. I'm planning to write a custom HTTP source for this. The source's `discover` method returns one stream per sensor so that I can filter which sensors I want to sync in the Airbyte UI. So, I have close to 1000 source streams. On the Postgres destination (using TimescaleDB), I obviously don't want to create 1000 tables corresponding to 1000 streams. Instead, I want all streams to be merged into a single table with the format (stream_name/sensor name, timestamp, value).
## Describe the solution you’d like
Ability to merge multiple source streams into a single destination table by adding the stream name as an additional column without having to write a custom destination in Airbyte, especially when all the streams have the same schema.
## Describe the alternative you’ve considered or used
Custom pipelines.
┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368215809205) by [Unito](https://www.unito.io)
コントリビューションガイド
評価
この issue はまだ評価されていません。