fluent / fluent/fluent-plugin-kafka
Timestamp is always parsed as local time
- Dominant language
- Ruby
- Stars
- 307
- Forks
- 178
- Avg merge
- 10h 37m
- Merged PRs (30d)
- 13
Description
We use UTC timestamps when we write logs in Kafka, and date partitioning on S3.
When I tried to parse a timestamp column like `{ "@t": "2020-11-11T07:58:58.7291622Z" }`, generated filename was `20201110_22xx` (GMT+9), not `20201111_07xx`!
`in-kafka-group` creates TimeParser with just `time_format`, since there is no parameter like `utc`.
https://github.com/fluent/fluent-plugin-kafka/blob/fe1d71e59ddfd941c37714b5bf6bd1d2d9b15a50/lib/fluent/plugin/in_kafka_group.rb#L134-L140
TimeParser uses local time by default. (`localtime = true`)
https://github.com/fluent/fluentd/blob/6757d70b7fdfcf5e4a6316b1ecd802dfeb0e326e/lib/fluent/time.rb#L208
Can we have `utc` parameter like parsers? When I set `time_source` as `kafka`, it worked properly. The timestamp was UTC.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.