Aiven-Open / Aiven-Open/cloud-storage-connectors-for-apache-kafka

Extend timestamp variable parameters in File name format

Open
#268 3 comments 0 reactions 1 assignee Claimed by @jeqo View on GitHub
S3
Dominant language
Java
Stars
58
Forks
39
Avg merge
2d 10h
Merged PRs (30d)
5

Description

# Scenario Overview
We use s3 filename template `prefix/'%Y_%m_%d__%H_%M_%S_%f` to sort filenames alphabetically.
The next new file is guaranteed to receive the following name in alphabetical order.

In kafka, we have several partitions of one topic, each of them must be written with the same prefix (prefix=topic_name) in order.
It's possible to ensure the files order with this template by running no more than 1 connector task.

# Issue:
Timestamp variable have next parameters:

```
unit parameter values:
yyyy - year, e.g. 2020 (please note that YYYY is deprecated and is interpreted as yyyy)
MM - month, e.g. 03
dd - day, e.g. 01
HH - hour, e.g. 24
```

# Consequences:
With these parameters, files recorded within 1 hour will not differ in name.
Adding the partition number and offset to the file name in the template can solve this problem, but it makes working with the root prefix more difficult.
Uniqueness can be ensured by adding minutes, seconds, milliseconds to the timestamp variable.

# Details:

Looks like it's enough to extend the following functionality :

```
private static final Map TIMESTAMP_FORMATTERS =
Map.of(
"yyyy", DateTimeFormatter.ofPattern("yyyy"),
"MM", DateTimeFormatter.ofPattern("MM"),
"dd", DateTimeFormatter.ofPattern("dd"),
"HH", DateTimeFormatter.ofPattern("HH")
);
```

with next parameters:

```
"%M" - Minutes in two-digit format.
"%S" - Seconds in two-digit format.
"%f" - Microseconds.
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.