airbytehq / airbytehq/airbyte

[destination-GCS] Overwrite does not work properly using variables in GCS Bucket Path

Open
#28,528 9 comments 14 reactions 0 assignees View on GitHub
area/connectors autoteam community connectors/destination/gcs team/use type/bug
Dominant language
Python
Stars
22.1k
Forks
5.3k
PR merge metrics
PR metrics pending

Description

### Connector Name

destination-google-cloud-storage

### Connector Version

0.4.4

### What step the error happened?

During the sync

### Revelant information

Hello,

The GCS destination does not properly apply the `Overwrite` strategy in certain circumstances.
I mean the existing files are not removed from GCS path before the sync, leading to duplicates files after the sync.

In GCS destination configuration settings, while filling `GCS Bucket Path` value:
- with a simple path like `my_folder/my_subfolder` : `Overwrite` strategy works as expected
- with a dynamic path like `my_folder/${STREAM_NAME}`: `Overwrite` strategy does not work as expected as explained above.

Reproducibility is really simple:
1. set up a GCS destination with simple path in `GCS Bucket Path` value
2. create a connection using this GCS destination (source does not matter) using `Overwrite` strategy
3. sync the connection => a file will be generated as expected
4. re-sync the connection => the file will be dropped first, then re-generated as expected
5. update the GCS destination `GCS Bucket Path` value with a dynamic path
6. re-sync the connection => the file will NOT be dropped first, then re-generated, leading to duplicates

Thank you for your help 👍

### Relevant log output

_No response_

### Contribute

- [ ] Yes, I want to contribute

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.