fluent / fluent/fluentd

time_key default needlessly overridden in json, regexp, and ltsv parsers

Open
#3,269 4 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

**Describe the bug**
The global default for `time_key` is `nil`, indicating that the event time should not be derived from the message at all, but the json, regexp, and ltsv parsers override this to `"time"`. Furthermore, the fluentd config syntax does not permit `time_key` to be set back to `nil`. as specifying `nil`, `null`, or `""` result in the literal string representations `"nil"`, `"null"`, and `""` being used. Specifying `"#{use_nil}"` results in an error that a value is required.

While it appears that `time_key` was required to be defined in the [original implementations](https://github.com/fluent/fluentd/commit/2f261882d4598af769a81a8e3056ecfb8734b7bf#diff-73e8d1657dbede482feb06b87014a3656632e7e507cd48fd7be2440aa848309e) of these parsers, it has not been required for some time now, and the result is an edge case where a message contains a field called `time` that is not parseable as time causes an exception and the message to be dropped.

It is possible to work around this issue by setting `time_key` to a value that you do not expect to occur as a key in your messages, but this is a kludge at best.

It's also worth noting here that the docs _do_ note that `time_key` is overridden by these parsers: https://docs.fluentd.org/configuration/parse-section#parse-parameters

**To Reproduce**
1. Define any input with a json parser.
2. Submit a message such as `{"time": {"begin": 1611709938020}}`

**Expected behavior**
The message to be decoded as a dict with a key named "time" and its arbitrary value.

**Environment**

- Fluentd or td-agent version: 1.11.5
- Operating system: docker image `fluent/fluentd:v1.11.5-1.0`

**Configuration**

```

# ...

@type json

```

**Error Log**

```
#1611709938020}(Hash)> @ /usr/lib/ruby/gems/2.7.0/gems/fluentd-1.11.5/lib/fluent/plugin/parser.rb:196:in `rescue in parse_time'
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.