[destination-GCS] Overwrite does not work properly using variables in GCS Bucket Path
- Dominant language
- Python
- Stars
- 22.1k
- Forks
- 5.3k
- PR merge metrics
- PR metrics pending
Description
### Connector Name
destination-google-cloud-storage
### Connector Version
0.4.4
### What step the error happened?
During the sync
### Revelant information
Hello,
The GCS destination does not properly apply the `Overwrite` strategy in certain circumstances.
I mean the existing files are not removed from GCS path before the sync, leading to duplicates files after the sync.
In GCS destination configuration settings, while filling `GCS Bucket Path` value:
- with a simple path like `my_folder/my_subfolder` : `Overwrite` strategy works as expected
- with a dynamic path like `my_folder/${STREAM_NAME}`: `Overwrite` strategy does not work as expected as explained above.
Reproducibility is really simple:
1. set up a GCS destination with simple path in `GCS Bucket Path` value
2. create a connection using this GCS destination (source does not matter) using `Overwrite` strategy
3. sync the connection => a file will be generated as expected
4. re-sync the connection => the file will be dropped first, then re-generated as expected
5. update the GCS destination `GCS Bucket Path` value with a dynamic path
6. re-sync the connection => the file will NOT be dropped first, then re-generated, leading to duplicates
Thank you for your help 👍
### Relevant log output
_No response_
### Contribute
- [ ] Yes, I want to contribute
Contributor guide
Assessment
This issue has not been assessed yet.