elastic / elastic/beats

Deprecation file logs should not emit data_stream.dataset

Open
#40,182 15 comments 3 reactions 1 assignee Claimed by @faec View on GitHub
>bug Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

### Elasticsearch Version

8.4.3

### Installed Plugins

_No response_

### Java Version

_bundled_

### OS Version

macos

### Problem Description

As per ECS, datastream.dataset is a constant-keyword: https://www.elastic.co/guide/en/ecs/master/ecs-data_stream.html#field-data-stream-dataset, which means, as per constant_keyword definition: the first document indexed, and having this field defined, will set this field for the remaining documents or makes documents with a different value to be rejected.

As a consequence, when user is doing stack monitoring with filebeat, indexing to the default filebeat-x.y.z datastream, any indexed document (coming from stack monitoring or any other filebeat modules) ends up with "data_stream.dataset":"deprecation.elasticsearch":

we should not emit data_stream.dataset in deprecation.json log file.
data_stream.datastream should still be emitted into `.logs-deprecation.elasticsearch-default` when cluster.deprecation_indexing.enabled: true (default)

the documentation of the `data_stream.dataset` field also indicates that it should have the same value as `event.dataset`. This makes me wonder if perhaps when logs are being ingested into ES cluster they should go into individual datastreams per data_stream.dataset value?

### Steps to Reproduce

emit deprecation logs

### Logs (if relevant)

_No response_

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.