Destination S3: long time to clean/delete s3 paths
- Lingua principale
- Python
- Stelle
- 22.1k
- Fork
- 5.3k
- Metriche di merge delle PR
- Metriche PR in attesa
Descrizione
### Connector Name
S3
### Connector Version
0.4.0
### What step the error happened?
During the sync
### Revelant information
90% of the connection execution time is used to clean s3 paths.
**Context:**
Source: Postgres
Destination: S3 Parquet
Extraction Method - Full refresh | Overwrite
Volume of data: 3GB
#Streams involved: 25
rows volume: 2_750_000
**Full refresh | Overwrite**
The connection takes to long to run when using this config. The connection spent 6h cleaning s3 paths and then the whole dataset was synced in 10 minutes
The connection longed the sames time even when the s3 paths were cleaned manually.
**Full refresh | Append**
The connection takes 10 minutes to run.
### Relevant log output
_No response_
### Contribute
- [X] Yes, I want to contribute
Guida per i contributori
Apri la guida per i contributori
Valutazione
Questa issue non è ancora stata valutata.