elastic / elastic/beats

Feature parity: Add a "dot_expander" Processor similar to what Ingest Pipelines have for merge conflict resolution

Open
#43,408 3 comments 0 reactions 0 assignees View on GitHub
needs_team
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Elastic Agent / Filebeat version: `8.17.1`

**Describe the enhancement:**
There is no Processor available in `libbeat` ([Filebeat](https://www.elastic.co/guide/en/beats/filebeat/current/defining-processors.html), [Elastic Agent](https://www.elastic.co/guide/en/fleet/current/elastic-agent-processor-configuration.html)) that provides similar functionality as is available to [Logstash's de-dot filter](https://www.elastic.co/guide/en/logstash/current/plugins-filters-de_dot.html) or [Ingest Pipeline's dot expander Processor](https://www.elastic.co/guide/en/elasticsearch/reference/current/dot-expand-processor.html
)

When pre-processing log files in JSON format at the edge host, the available [decode_json_fields Processor](https://www.elastic.co/guide/en/fleet/current/decode-json-fields.html) _does_ offer a **expand_keys** option:

> Whether keys in the decoded JSON should be recursively de-dotted and expanded into a hierarchical object structure. For example, {"a.b.c": 123} would be expanded into {"a":{"b":{"c":123}}}

But [the source code](https://github.com/elastic/beats/blob/v8.17.1/libbeat/common/jsontransform/expand.go#L41-L48) for that expansion states that **_any_ conflicts are errors**:
```
// expandFields de-dots the keys in m by expanding them in-place into a
// nested object structure, merging objects as necessary. If there are any
// conflicts (i.e. a common prefix where one field is an object and another
// is a non-object), an error will be returned.
//
// Note that expandFields is destructive, and in the case of an error the
// map may be left in a semi-expanded state.
func expandFields(m mapstr.M) error {
```

Unlike the Ingest Pipeline's `dot_expander` Processor which provides options for conflict resolution:
> override
Controls the behavior when there is already an existing nested object that conflicts with the expanded field. When false, the processor will merge conflicts by combining the old and the new values into an array. When true, the value from the expanded field will overwrite the existing value.

**Describe a specific use case for the enhancement or feature:**
We encountered a scenario where we are replacing the [Elastic Docker Log Driver](https://www.elastic.co/guide/en/beats/loggingplugin/current/log-driver-overview.html) with Elastic Agent using a [Docker integration](https://www.elastic.co/guide/en/integrations/current/docker.html).

The Log Driver was configured to invoke a custom pipeline
```
version: "2.4"
services:
# .. SNIP
logging:
driver: "elastic"
options:
# .. SNIP
pipeline: "docker_log_driver_pipeline"
```
Where that pipeline used the `dot_expander` Processor. Now that Elastic Agent is taking over we are trying to define Processors on the Integration to move a lot of this pre-processing to the edge.

It just so happened that an application was writing [Elastic Common Schema](https://www.elastic.co/guide/en/ecs/current/index.html) compliant logs including the `service.*` fields like this
```
{
...
"service": {
"name": "service_name",
"environment": "qa",
"version": "1.0.0"
}
}
```

Sometime later [APM](https://www.elastic.co/guide/en/apm/agent/nodejs/current/configuration.html) was introduced to the application:

```
const baseConfig: elasticapm.AgentConfigOptions = {
environment: env.NODE_ENV,
serviceName: 'service_name',
serviceNodeName: serverNodeName,
serviceVersion: env.VERSION
};

export const apm = elasticapm.start({
...baseConfig,
apiKey: env.ELASTIC_APM_API_KEY,
serverUrl: env.ELASTIC_APM_SERVER_URL
})
```

The application is logging with Pino and utilizing Elastic's ECS Logger for Pino which [automatically decorates logs with data from APM](https://www.elastic.co/guide/en/ecs-logging/nodejs/current/pino.html#pino-apm)

> ...then a number of fields are added to log records to correlate between APM services or traces and logging
data: ... * A number of service identifier fields determined by or configured on the APM agent allow cross-linking between services and logs in Kibana — service.name, service.version, service.environment, service.node.name.

So what ends up happening is that we end up with log JSON containing _both_ dotted fields (from APM) _and_ Object fields (from application):
```
{
...
"service.name": "service_name",
"service.environment": "qa",
"service.version": "1.0.0",
"service.node.name": "qa77",
"service": {
"name": "service_name",
"environment": "qa",
"version": "1.0.0"
}
}
```
This wasn't a problem for the Docker Log Driver with the Ingest Pipeline using `dot_expander`.

When the Elastic Agent's Integration is given a Processor definition:
```
- decode_json_fields:
fields: ["message"]
process_array: false
max_depth: 1
target: ""
overwrite_keys: true
add_error_key: true
expand_keys: false # true can blow up in cases that service metadata is included in both dotted (APM) and Object form (app)
```
Setting `expand_keys: true` will cause failures and the Processor will add error information to the log event because of `add_error_key: true` :
```
"error": {
"message": "cannot expand \"service.version\": found existing (string) value",
"type": "json"
},
```

**Conclusion**
So the only way to successfully process the JSON at the edge is for `expand_keys: false` but then we have a situation where we _still_ want to dedot/dot_expand but cannot without either
* A new feature in `decode_json_fields` to allow conflict resolution
* A new `dot_expander` Processor usable by Elastic Agent & Beats
* Employing an Ingest Pipeline dot_expander Processor anyway - which ideally should have been performed at the edge host instead
* Make a JavaScript implementation of [the dot_expander Processor](https://github.com/elastic/elasticsearch/blob/v8.17.1/modules/ingest-common/src/main/java/org/elasticsearch/ingest/common/DotExpanderProcessor.java) as a script Processor in Elastic Agent - might work but we shouldn't have to do this.

Anyway, in conclusion, `libbeat` could use a standalone `dot_expander` Processor of its own that doesn't depend on the `decode_json_fields` Processor.

Contributor guide

Open the contributing guide

Research direction

Start by reading libbeat/common/jsontransform/expand.go, especially expandFields, and compare its conflict behavior with the referenced Ingest Pipeline dot_expander. The work is done when Beats and Elastic Agent have a standalone dot_expander Processor that handles dotted-field conflicts according to the requested behavior; the issue does not name a test file.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.