fluent / fluent/fluent-plugin-kafka

Timestamp is always parsed as local time

Open
#377 1 comment 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
307
Forks
178
Avg merge
10h 37m
Merged PRs (30d)
13

Description

We use UTC timestamps when we write logs in Kafka, and date partitioning on S3.
When I tried to parse a timestamp column like `{ "@t": "2020-11-11T07:58:58.7291622Z" }`, generated filename was `20201110_22xx` (GMT+9), not `20201111_07xx`!

`in-kafka-group` creates TimeParser with just `time_format`, since there is no parameter like `utc`.

https://github.com/fluent/fluent-plugin-kafka/blob/fe1d71e59ddfd941c37714b5bf6bd1d2d9b15a50/lib/fluent/plugin/in_kafka_group.rb#L134-L140

TimeParser uses local time by default. (`localtime = true`)
https://github.com/fluent/fluentd/blob/6757d70b7fdfcf5e4a6316b1ecd802dfeb0e326e/lib/fluent/time.rb#L208

Can we have `utc` parameter like parsers? When I set `time_source` as `kafka`, it worked properly. The timestamp was UTC.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.