airbytehq / airbytehq/airbyte

Support wildcard expression to load multiple files from GCS

Aberta
#2,512 6 comentários 0 reações 0 responsáveis Ver no GitHub
autoteam connectors/source/google-cloud-storage Icebox team/extensibility team/marketplace type/enhancement
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

## Tell us about the problem you're trying to solve
We want to load files on GCS with wildcard expression, say about `gs://my-bucket/folder/data/data*.csv`, because it is quite common to use multiple files to express single data.

## Describe the solution you’d like
It seems that `URLFile` seems to accept only a single file. I am not sure the best solution. We might have to change the specification to deal with file(s).
https://github.com/airbytehq/airbyte/blob/d4267818ae0f81f584f856b6b208ee76d774e15a/airbyte-integrations/connectors/source-file/source_file/client.py#L51

## Describe the alternative you’ve considered or used
Technically, we might be able to read files one by one. But, it is very annoying.

┆Issue is synchronized with this [Asana task](https://app.asana.com/0/1200367912513076/1200368096116135) by [Unito](https://www.unito.io)

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.