elastic / elastic/beats

Monitoring indices map cgroup memory metrics as keyword instead of numeric

Open
#51,637 1 comment 0 reactions 0 assignees View on GitHub
Team:Elastic-Agent-Data-Plane
Dominant language
Go
Stars
12.7k
Forks
5k
Avg merge
2d 15m
Merged PRs (30d)
385

Description

Several cgroup memory metrics collected by Elastic Cloud Enterprise (ECE) 4.0.3 monitoring are indexed as `keyword` fields instead of numeric types.

As a result, these metrics cannot be used for:

* Lens visualizations
* TSVB
* ES|QL numeric operations
* aggregations
* sorting
* threshold-based alerting

This makes it difficult to monitor container memory usage and memory limits, despite the values themselves being numeric.

## Affected fields

Examples include:

* `memory.limit.bytes`
* `memory.usage.bytes`

(There may be additional cgroup memory metrics affected.)

## Current mapping

Example:

```json
"memory": {
"properties": {
"limit": {
"properties": {
"bytes": {
"type": "keyword"
}
}
},
"usage": {
"properties": {
"bytes": {
"type": "keyword"
}
}
}
}
}
```

## Expected mapping

These fields should be mapped as numeric values, for example:

```json
"type": "long"
```

since they represent byte counts.

## Why this matters

Memory limit and usage are fundamental infrastructure metrics that operators commonly use to:

* monitor memory consumption over time
* calculate memory utilization percentages
* create threshold alerts
* build dashboards
* correlate memory pressure with CPU throttling and other container metrics

When stored as `keyword`, these common observability workflows are unnecessarily limited.

## Reproduction

1. Enable ECE monitoring.
2. Inspect the mapping of the monitoring indices.
3. Observe that cgroup memory byte fields are mapped as `keyword`.
4. Attempt to build a Lens visualization or perform numeric aggregations on these fields.

## Additional context

A discussion was started on the Elastic forums where it was suggested that this should be tracked as a GitHub issue.

Forum discussion:
[https://discuss.elastic.co/t/ece-monitoring-cgroup-memory-fields-mapped-as-keyword-instead-of-numeric/386861](https://discuss.elastic.co/t/ece-monitoring-cgroup-memory-fields-mapped-as-keyword-instead-of-numeric/386861)

## Expected outcome

Map cgroup memory byte metrics using an appropriate numeric type (e.g. `long`) so they can be used consistently throughout the Elastic Stack for visualization, alerting, ES|QL, and analytics.

Contributor guide

Open the contributing guide

Research direction

Start by locating the monitoring index mapping responsible for the cgroup fields named in the issue, especially memory.limit.bytes and memory.usage.bytes. Inspect how their types are assigned, then verify the affected mappings use a numeric type and that the monitoring index supports numeric aggregations and sorting.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
observability-sre
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.