airbytehq / airbytehq/airbyte

Destination S3: long time to clean/delete s3 paths

Aberta
#26,525 4 comentários 1 reação 0 responsáveis Ver no GitHub
area/connectors community connectors/destination/s3 connectors/destinations-files frozen releaseStage/ga team/destinations type/bug
Linguagem predominante
Python
Estrelas
22.1k
Forks
5.4k
Merge médio
5h
PRs com merge (30d)
671

Descrição

### Connector Name

S3

### Connector Version

0.4.0

### What step the error happened?

During the sync

### Revelant information

90% of the connection execution time is used to clean s3 paths.

**Context:**

Source: Postgres
Destination: S3 Parquet
Extraction Method - Full refresh | Overwrite
Volume of data: 3GB
#Streams involved: 25
rows volume: 2_750_000

**Full refresh | Overwrite**

The connection takes to long to run when using this config. The connection spent 6h cleaning s3 paths and then the whole dataset was synced in 10 minutes

The connection longed the sames time even when the s3 paths were cleaned manually.

**Full refresh | Append**

The connection takes 10 minutes to run.

### Relevant log output

_No response_

### Contribute

- [X] Yes, I want to contribute

Guia de contribuição

Abrir o guia de contribuição

Avaliação

Esta issue ainda não foi avaliada.

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.