airbytehq / airbytehq/airbyte

Support wildcard expression to load multiple files from GCS

Aperta
#2,512 6 commenti 0 reazioni 0 assegnatari Vedi su GitHub
autoteam connectors/source/google-cloud-storage Icebox team/extensibility team/marketplace type/enhancement
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

## Tell us about the problem you're trying to solve
We want to load files on GCS with wildcard expression, say about `gs://my-bucket/folder/data/data*.csv`, because it is quite common to use multiple files to express single data.

## Describe the solution you’d like
It seems that `URLFile` seems to accept only a single file. I am not sure the best solution. We might have to change the specification to deal with file(s).
https://github.com/airbytehq/airbyte/blob/d4267818ae0f81f584f856b6b208ee76d774e15a/airbyte-integrations/connectors/source-file/source_file/client.py#L51

## Describe the alternative you’ve considered or used
Technically, we might be able to read files one by one. But, it is very annoying.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368096116135) by [Unito](https://www.unito.io)

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.