Separate internal from custom application metricsets
- Dominant language
- Gherkin
- Stars
- 427
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
In 8.3 APM Server will begin storing well-defined metrics, like `system.memory.total` and `jvm.memory.heap.used` in `metrics-apm.internal-` (https://github.com/elastic/apm-server/issues/7520). Previously, these metrics would be stored in `metrics-apm.app.-`, which led to memory issues in Elasticsearch at scale.
The `internal` metrics data stream uses strict mapping, whereas the service-specific metric data streams rely on dynamic mapping. Thus, we must ensure no unknown metrics are indexed into the internal metrics data stream.
To make all this happen, APM Server now has a hard coded list of metrics defined: https://github.com/elastic/apm-server/tree/main/apmpackage/apm/data_stream/internal_metrics/fields. When APM Server receives a metricset, it will look at all the metrics defined within it: if they are all well-defined, then the metricset is sent to the internal data stream; otherwise it is sent to a service-specific data stream. Thus, if an agent begins sending a new metric, APM Server will begin sending the entire metricset to a service-specific data stream.
I propose that we instead give agents the ability to inform APM Server that a metricset is intended to contain only "internal" metrics, i.e. metrics defined by the agent, and not by the application. APM Server would then discard any unknown metrics from internal metricsets, and send the remainder to the internal metrics data stream. This has been POC'd in https://github.com/elastic/apm-server/pull/8064. Without agent changes, the behaviour would the same as in 8.3.
**8.3**
- If an agent sends a metricset with `system.memory.total`, then APM Server will index that metricset into `metrics-apm.internal-`
- If an agent sends a metricset with `system.memory.total` and `foo.bar.baz`, then APM Server will index that metricset into `metrics-apm.app.-` due to the unknown metric
**Proposal**
- If an agent sends metricset without `internal` defined, with `system.memory.total` and `foo.bar.baz`, then APM Server will index that metricset into `metrics-apm.app.-` due to the unknown metric
- If an agent sends a `internal: true` metricset with `system.memory.total` and `foo.bar.baz`, then APM Server will discard `foo.bar.baz` and index the metricset with just `system.memory.total` into `metrics-apm.internal-`
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.