fluent / fluent/fluentd

Make it possible to Specify ltsv parser plugin quoted value

Open
#3,575 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Ruby
Stars
13.6k
Forks
1.4k
Avg merge
1d 3h
Merged PRs (30d)
20

Description

### Is your feature request related to a problem? Please describe.

LTSV plugin can't understand quoted values
Example:
Having log:
`level=warn ts=2021-12-20T05:56:00.397096942Z caller=operator.go:516 component=alertmanageroperator msg="alertmanager key=kube-system/prometheus-operator-kube-s-alertmanager, field spec.baseImage is deprecated, 'spec.image' field should be used instead"`

using ltsv plugin with next settings:
```

@type ltsv
delimiter_pattern /\s+/
label_delimiter =
time_key ts
time_format %Y-%m-%dT%H:%M:%S.%N%Z

```

expected json:
```
{
"level": "warn",
"ts": "2021-12-20T05:56:00.397096942Z",
"caller": "operator.go:516",
"component": "alertmanageroperator",
"msg": "alertmanager key=kube-system/prometheus-operator-kube-s-alertmanager, field spec.baseImage is deprecated, 'spec.image' field should be used instead"
}
```

actual result:
```
{
"level": "warn",
"ts": "2021-12-20T05:56:00.397096942Z",
"caller": "operator.go:516",
"component": "alertmanageroperator",
"msg": "\"alertmanager",
"key": "kube-system/prometheus-operator-kube-s-alertmanager,"
}
```
**msg** field is truncated, wrong **key** field is added to result json.
Actual result is parsed wrong, because it doesn't understand double quotes where all **msg** field value enclosed.

### Describe the solution you'd like

LTSV plugin understands, that value after **label_delimiter** could be enclosed in some symbol sequence, for example, double or single quotes. It will help to solve such cases.

### Describe alternatives you've considered

It made me to use **regexp** parser type and put everything, that goes after **ts** field as message with next expression:
`/level=(?.*)\sts=(?\d{4}-\d{2}-\d{2}T\d{2}:\d{2}:\d{2}\.\d{3,}[A-Z]+) (?.*)/`
which is much worse solution, that can be done with proper ltsv plugin implementation.

### Additional context

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.