[destination-GCS] Overwrite does not work properly using variables in GCS Bucket Path
- Linguagem predominante
- Python
- Estrelas
- 22.1k
- Forks
- 5.4k
- Merge médio
- 5h
- PRs com merge (30d)
- 671
Descrição
### Connector Name
destination-google-cloud-storage
### Connector Version
0.4.4
### What step the error happened?
During the sync
### Revelant information
Hello,
The GCS destination does not properly apply the `Overwrite` strategy in certain circumstances.
I mean the existing files are not removed from GCS path before the sync, leading to duplicates files after the sync.
In GCS destination configuration settings, while filling `GCS Bucket Path` value:
- with a simple path like `my_folder/my_subfolder` : `Overwrite` strategy works as expected
- with a dynamic path like `my_folder/${STREAM_NAME}`: `Overwrite` strategy does not work as expected as explained above.
Reproducibility is really simple:
1. set up a GCS destination with simple path in `GCS Bucket Path` value
2. create a connection using this GCS destination (source does not matter) using `Overwrite` strategy
3. sync the connection => a file will be generated as expected
4. re-sync the connection => the file will be dropped first, then re-generated as expected
5. update the GCS destination `GCS Bucket Path` value with a dynamic path
6. re-sync the connection => the file will NOT be dropped first, then re-generated, leading to duplicates
Thank you for your help 👍
### Relevant log output
_No response_
### Contribute
- [ ] Yes, I want to contribute
Guia de contribuição
Avaliação
Esta issue ainda não foi avaliada.