elastic / elastic/integrations

[kubernetes_otel] "View pod logs in discover" does not show logs of ecs pods.

Open
#19,173 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Handlebars
Stars
333
Forks
647
Avg merge
3d 4h
Merged PRs (30d)
209

Description

Image

Image

Since the filter uses the otel semconv field `k8s.pod.name` to look up pods, only logs of otel data_streams are shown. Thus, logs from pods which are ingested via classic ecs integration package pipelines are not displayed. Is this the intended behavior?

From my point of view, the elefant in the room may be the confusing amount of different naming schemes we see at the moment:

| Namespace | Comment |
| ---------------------------------- | ----------------------------------------------------------------------------------------------------------------------------- |
| orchestrator | Original ECS namespace for normalized orchestrator fields, like from kubernetes |
| resource.attributes.k8s | Fields used by OTEL k8s semconv, flattened fields, some of them mapped as keyword |
| attributes.k8s | Fields used by OTEL k8s semconv, flattened fields |
| k8s | Passthrough fields of mapped OTEL resource.attributes.k8s, attributes.k8s |
| kubernetes | Used in ECS integrations, OTEL collector elasticsearch exporter, aliased from k8s fields |

Some fields are "real" fields from source, others are passthrough fields, yet others aliases. Data is ingested through
different pipelines, i.e. native OTLP, OTEL elasticsearch exporters, elastic agents in ECS, etc. The result is a set of
differently mapped data_streams which are hard to correlate. On top of that there is a set of query languages,
each with it's pro's and con's. The one to solve it all among them ES|QL, which seems like the new standard, but
still has some limitations and strange behavior when used in discover in combination with otel data_streams.

Long story short, at the moment it is hard to keep the overview on all those different fields and mappings and which is the best way to analyse and correlate data from different sources. I would highly appreciate if there would be some kind
of guideline which helps to understand on which fields we can trust and a clear path of migration towards OTEL semconv, if this is the idea. If this is the wrong place to discuss this, please let me know where I should address it correctly.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.