elastic / elastic/apm

Separate internal from custom application metricsets

Open
#647 7 comments 0 reactions 0 assignees View on GitHub
discussion
Dominant language
Gherkin
Stars
427
Forks
125
PR merge metrics
No merged PRs in 30d

Description

In 8.3 APM Server will begin storing well-defined metrics, like `system.memory.total` and `jvm.memory.heap.used` in `metrics-apm.internal-` (https://github.com/elastic/apm-server/issues/7520). Previously, these metrics would be stored in `metrics-apm.app.-`, which led to memory issues in Elasticsearch at scale.

The `internal` metrics data stream uses strict mapping, whereas the service-specific metric data streams rely on dynamic mapping. Thus, we must ensure no unknown metrics are indexed into the internal metrics data stream.

To make all this happen, APM Server now has a hard coded list of metrics defined: https://github.com/elastic/apm-server/tree/main/apmpackage/apm/data_stream/internal_metrics/fields. When APM Server receives a metricset, it will look at all the metrics defined within it: if they are all well-defined, then the metricset is sent to the internal data stream; otherwise it is sent to a service-specific data stream. Thus, if an agent begins sending a new metric, APM Server will begin sending the entire metricset to a service-specific data stream.

I propose that we instead give agents the ability to inform APM Server that a metricset is intended to contain only "internal" metrics, i.e. metrics defined by the agent, and not by the application. APM Server would then discard any unknown metrics from internal metricsets, and send the remainder to the internal metrics data stream. This has been POC'd in https://github.com/elastic/apm-server/pull/8064. Without agent changes, the behaviour would the same as in 8.3.

**8.3**

- If an agent sends a metricset with `system.memory.total`, then APM Server will index that metricset into `metrics-apm.internal-`
- If an agent sends a metricset with `system.memory.total` and `foo.bar.baz`, then APM Server will index that metricset into `metrics-apm.app.-` due to the unknown metric

**Proposal**

- If an agent sends metricset without `internal` defined, with `system.memory.total` and `foo.bar.baz`, then APM Server will index that metricset into `metrics-apm.app.-` due to the unknown metric
- If an agent sends a `internal: true` metricset with `system.memory.total` and `foo.bar.baz`, then APM Server will discard `foo.bar.baz` and index the metricset with just `system.memory.total` into `metrics-apm.internal-`

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.