Deprecation file logs should not emit data_stream.dataset
- Dominant language
- Go
- Stars
- 12.7k
- Forks
- 5k
- Avg merge
- 2d 15m
- Merged PRs (30d)
- 385
Description
### Elasticsearch Version
8.4.3
### Installed Plugins
_No response_
### Java Version
_bundled_
### OS Version
macos
### Problem Description
As per ECS, datastream.dataset is a constant-keyword: https://www.elastic.co/guide/en/ecs/master/ecs-data_stream.html#field-data-stream-dataset, which means, as per constant_keyword definition: the first document indexed, and having this field defined, will set this field for the remaining documents or makes documents with a different value to be rejected.
As a consequence, when user is doing stack monitoring with filebeat, indexing to the default filebeat-x.y.z datastream, any indexed document (coming from stack monitoring or any other filebeat modules) ends up with "data_stream.dataset":"deprecation.elasticsearch":
we should not emit data_stream.dataset in deprecation.json log file.
data_stream.datastream should still be emitted into `.logs-deprecation.elasticsearch-default` when cluster.deprecation_indexing.enabled: true (default)
the documentation of the `data_stream.dataset` field also indicates that it should have the same value as `event.dataset`. This makes me wonder if perhaps when logs are being ingested into ES cluster they should go into individual datastreams per data_stream.dataset value?
### Steps to Reproduce
emit deprecation logs
### Logs (if relevant)
_No response_
Contributor guide
Assessment
This issue has not been assessed yet.