Destination S3: choose what Data to delete when using Overwrite Sync Mode
- Lenguaje dominante
- Python
- Estrellas
- 22.1k
- Forks
- 5.3k
- Métricas de merge de PR
- Métricas de PR pendientes
Descripción
## Tell us about the problem you're trying to solve
Hello Airbyte Team,
I habe am trying to sync some data to our S3-Destination.
I have my path configured as:

and my path format as:

What I am trying to achieve is, that when using the Overwrite Sync-Mode, all data under the subpath from my specific _snapshot_date is deleted. What currently happens is, that airbyte is deleting everything under `/raw/nomip/${STREAM_NAME}/`
which makes sense on how the files to delete are determined.
S3StorageOperations.java:
```java
protected String getRegexFormat(final String namespace, final String streamName, final String pathFormat) {
final String namespaceStr = nameTransformer.getNamespace(isNotBlank(namespace) ? namespace : "");
final String streamNameStr = nameTransformer.getIdentifier(streamName);
return nameTransformer.applyDefaultCase(pathFormat
.replaceAll(Pattern.quote(FORMAT_VARIABLE_NAMESPACE), namespaceStr)
.replaceAll(Pattern.quote(FORMAT_VARIABLE_STREAM_NAME), streamNameStr)
.replaceAll(Pattern.quote(FORMAT_VARIABLE_YEAR), "[0-9]{4}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_MONTH), "[0-9]{2}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_DAY), "[0-9]{2}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_HOUR), "[0-9]{2}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_MINUTE), "[0-9]{2}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_SECOND), "[0-9]{2}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_MILLISECOND), "[0-9]{4}")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_EPOCH), "[0-9]+")
.replaceAll(Pattern.quote(FORMAT_VARIABLE_UUID), ".*")
.replaceAll("/+", "/")
// match part_id and extension at the end
+ ".*");
}
```
The STREAM_NAME gets replaced while every other component is just replaced by a regex.
Our problem is, that we want to enforce to only have 1 sync per day with the newest data. Usually this works because we use a regex that runs only one a day, but if someone is staring a manual run, the previous data must be deleted by hand (or external script.)
## Describe the solution you’d like
It would be nice to use the placeholders (at least year, month, day) to define a fixed path under that every old sync gets deleted.
## Are you willing to submit a PR?
Maybe, but a workaround would be sufficient for now if possible.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.