[out_syslog] support nested keys in syslog output plugin
- Dominant language
- C
- Stars
- 8.1k
- Forks
- 2k
- Avg merge
- 4d 20h
- Merged PRs (30d)
- 71
Description
**Is your feature request related to a problem? Please describe.**
Our organisation requires processing records with [nested data](https://docs.fluentbit.io/manual/pipeline/filters/nest). We also need to stream records to a syslog server, which is newly supported in the master branch thanks to PR #1601 being merged. However, this feature does not support using nested keys (e.g. `"log"=>"syslog"=>"severity"=>"code"` when [using ECS](https://www.elastic.co/guide/en/ecs/1.5/ecs-log.html)). The current syntax is as follows:
```ini
[OUTPUT]
Name syslog
<...>
Syslog_Severity_key my_severity_key
```
This does not allow us to handle a record structured as follows ([ECS 1.5](https://www.elastic.co/guide/en/ecs/current/ecs-field-reference.html), very simplified):
```json
{
"log": {
"syslog": {
"severity": {
"name": "warning",
"code": "4"
}
}
},
"message": "my very important message"
}
```
**Describe the solution you'd like**
Ideally, the syslog output plugin should support nested keys using the current interface. This could be done by adding a new plugin parameter (e.g. `NestedKeySeparator`) that if provided, would tell the plugin to split all key strings into nested key sequences. This means that internally, the native format for the keys would become arrays which are used to iterate over nested record keys.
For example, we could parse the JSON message above using the following config:
```ini
[OUTPUT]
Name syslog
<...>
Syslog_Severity_key log.syslog.severity.code
NestedKeySeparator .
```
**Describe alternatives you've considered**
We have examined the possibility of adding a `filter` step that lifts the records before they are sent with `out_syslog`. However, this has the following drawbacks:
1. Additional `filter` step introduced in the processing pipeline
2. Causes the record to no longer respect our schema, which interferes with other outputs in the same pipeline
Point (1) is basically negligible in our use cases. However, point (2) is a very major drawback: if we want to route the same processed event to e.g. Elasticsearch and syslog, we would either need to break our Elasticsearch schema in order to use top-level keys for syslog, _or_ we would need to duplicate our pipeline for non-syslog outputs.
**Additional context**
Our use case is to use Fluent Bit to read a single input (e.g. system journal), process the events into a structured format, and send those processed events off to Elasticsearch, syslog and potentially other FluentD/Fluent Bit instances. The current behaviour will prevent us from relying on the official syslog output plugin and potentially force us into maintaining our own output plugin, which is obviously undesireable when 99.9% of the features we need are present in the current implementation.
I would appreciate any input on the solution proposed in this issue. For what it's worth, we are willing to create a pull request with this feature if the community is open to it.
Contributor guide
Assessment
This issue has not been assessed yet.