airbytehq / airbytehq/airbyte

Mapping multiple source streams into a single destination table on Postgres

Aberta
#2,224 6 comentários 3 reações 0 responsáveis Ver no GitHub
connectors/destination/postgres connectors/destinations-database frozen lang/java normalization team/destinations transformation type/enhancement
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

## Tell us about the problem you're trying to solve
I have a 3rd party API from which I read industrial sensor data (time-series data) of the form (timestamp, value) for each of the close to 1000 sensors. I'm planning to write a custom HTTP source for this. The source's `discover` method returns one stream per sensor so that I can filter which sensors I want to sync in the Airbyte UI. So, I have close to 1000 source streams. On the Postgres destination (using TimescaleDB), I obviously don't want to create 1000 tables corresponding to 1000 streams. Instead, I want all streams to be merged into a single table with the format (stream_name/sensor name, timestamp, value).

## Describe the solution you’d like
Ability to merge multiple source streams into a single destination table by adding the stream name as an additional column without having to write a custom destination in Airbyte, especially when all the streams have the same schema.

## Describe the alternative you’ve considered or used
Custom pipelines.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368215809205) by [Unito](https://www.unito.io)

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.