airbytehq / airbytehq/airbyte

Destination S3: long time to clean/delete s3 paths

Aperta
#26,525 4 commenti 1 reazione 0 assegnatari Vedi su GitHub
area/connectors community connectors/destination/s3 connectors/destinations-files frozen releaseStage/ga team/destinations type/bug
Lingua principale
Python
Stelle
22.1k
Fork
5.3k
Metriche di merge delle PR
Metriche PR in attesa

Descrizione

### Connector Name

S3

### Connector Version

0.4.0

### What step the error happened?

During the sync

### Revelant information

90% of the connection execution time is used to clean s3 paths.

**Context:**

Source: Postgres
Destination: S3 Parquet
Extraction Method - Full refresh | Overwrite
Volume of data: 3GB
#Streams involved: 25
rows volume: 2_750_000

**Full refresh | Overwrite**

The connection takes to long to run when using this config. The connection spent 6h cleaning s3 paths and then the whole dataset was synced in 10 minutes

The connection longed the sames time even when the s3 paths were cleaned manually.

**Full refresh | Append**

The connection takes 10 minutes to run.

### Relevant log output

_No response_

### Contribute

- [X] Yes, I want to contribute

Guida per i contributori

Apri la guida per i contributori

Valutazione

Questa issue non è ancora stata valutata.

Ricevi le nuove issue nella tua casella

Un breve riepilogo di issue GitHub adatte ai principianti.