fluent / fluent/fluent-bit

Add remove_keys_except parameter to Loki output

Open
#8,892 14 comments 7 reactions 0 assignees View on GitHub
Dominant language
C
Stars
8.1k
Forks
2k
Avg merge
4d 20h
Merged PRs (30d)
71

Description

**Is your feature request related to a problem? Please describe.**

We use Fluent Bit to collect and ship logs to Loki. The input parsers and the various filters in the pipeline all inject various metadata into the record which we use to construct a `label_keys` stanza to populate the Loki log stream labels from that metadata.

We then need to ensure the log message is delivered to Loki exactly as the application has logged it, so in the `loki` output we set `drop_single_key=true` and `line_format=key_value`, and then need to explicitly drop (using `remove_keys`) all fields in the record *other* than the original application log message (as determined by the input parser).

By the time all parsers and filters run, the record contains quite a number of fields from the accumulated metadata. This requires us to know a priori all possible keys, so we can drop all but the key containing the original log message.

Problematically, with the `ec2` filter and `tags_enabled`, we can't reliably know what those keys are. The `ec2` filter drops all EC2 instance tags directly at the root of the record (unlike, say, the `kubernetes` filter which creates a `kubernetes` map within the record), which means we can't use `remove_keys` because we don't always know every possible tag that might exist on the EC2 instance.

Here's a more concrete example of the pipeline portion of Fluent Bit configuration intended for AWS ECS on EC2 (host-level collection, not FireLens):

```yaml
pipeline:
inputs:
- name: tail
tag: tail.containers.*
path: /var/lib/docker/containers/*/*-json.log
storage.type: filesystem
parser: docker
filters:
- name: aws
match: "*"
az: true
ec2_instance_id: true
hostname: true
tags_enabled: true
- name: ecs
match: tail.containers.*
ecs_tag_prefix: tail.containers.var.lib.docker.containers.
add:
- task $TaskID
- cluster $ClusterName
- container $ECSContainerName
outputs:
- name: loki
match: "*"
host: loki.example.com
port: 443
line_format: key_value
drop_single_key: true
remove_keys:
myorg_environment, myorg_service, myorg_component,
cluster, task, container, attrs, ec2_instance_id, az
labels:
service=${FLB_SERVICE}, service=$myorg_service, service=$attrs['myorg_service'],
environment=${FLB_ENVIRONMENT}, environment=$myorg_environment, environment=$attrs['myorg_environment'],
component=${FLB_COMPONENT}, component=$myorg_component, component=$attrs['myorg_component'],
cluster=$cluster, task=$task, container=$container, host=$ec2_instance_id, az=$az
```

In this example, we set various Loki labels first, in some cases, via host-level environment variables (e.g. `${FLB_SERVICE}`), then, if it exists, using an EC2 instance tag (e.g. `myorg_service`), and then, if it exists (and this would take priority), a docker label specified in the ECS task definition (e.g. `myorg_service`).

As you can see, in order to preserve the original log line, `remove_keys` needs to explicitly list everything expected. When the EC2 instance contains tags not in that list, they aren't removed from the record, and they are shipped to Loki.

**Describe the solution you'd like**

I'd like to replace `remove_keys` in the example above with just this:

```yaml
remove_keys_except: log
```

Which would discard all keys from the record except `log` to ensure what's sent to Loki as the log event is only what the application actually logged, without extraneous FluentBit-collected metadata.

**Describe alternatives you've considered**

A Lua filter can accomplish this too, and that's what we're doing.

But a native option would produce clearer configuration and would be more performant.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.