Normalize metrics reported based on cgroup in all APM agents
- Dominant language
- Gherkin
- Stars
- 427
- Forks
- 125
- PR merge metrics
- No merged PRs in 30d
Description
# Description
## Problem
Currently all our APM Agents report memory and CPU usage on some way. When the application runs directly on a physical or virtual machine, metrics reported are typically correct.
However, our metric-story when the monitored application runs within a container and we report memory/CPU usage based on cgroups is not that clear.
Specific issues:
1. Currently most agents send memory metrics based on cgroups (`system.process.cgroup.memory.mem.limit.bytes` and `system.process.cgroup.memory.mem.usage.bytes`), but not all of them send CPU related metrics based on cgroups (e.g. Java does - it uses an API which takes container CPU usage into consideration). There are 2 problems with this: 1) our charts will be inconsistent: we may chart memory usage from the point of a container but chart CPU usage from the host's point of view and 2) APM Agents may report different CPU usage compared to other components - e.g. we know that metricbeat and some APM Agents report different CPU usage in Kubernetes. [Here](https://www.elastic.co/guide/en/beats/metricbeat/current/exported-fields-system.html#_stats_12) is a list of Kubernetes related CPU usage metrics.
2. The UI team currently does a [fairly complex calculation](https://github.com/elastic/kibana/blob/05aa070bca05f2df837a4ac5f38c176498b28100/x-pack/plugins/apm/server/routes/metrics/by_agent/shared/memory/index.ts#L59-L103) to calculate the total memory usage - the complexity here is added again by memory related metrics based on cgroup. The key issue is in `percentCgroupMemoryUsedScript` - the `system.process.cgroup.memory.mem.limit.bytes` may not be set, which means a pod can take up all the memory of the host. But in this case agents typically send the "magic value" of [`9223372036854771712L`](https://github.com/elastic/kibana/blob/05aa070bca05f2df837a4ac5f38c176498b28100/x-pack/plugins/apm/server/routes/metrics/by_agent/shared/memory/index.ts#L90), which is then "fixed" by the UI. While this works for our UI, for a user it's almost impossible to recreate a correct memory usage graph when an APM Agent sends memory related metrics based on cgroups.
## Solution
1. For the issue 1 above, each APM Agent must make sure that if the application runs within a container, the CPU usage reported is from the POD's point of view. So `system.cpu.total.norm.pct` and `system.process.cpu.total.norm.pct` should be aligned with the reported cgroup memory metrics. For some agents this may be already implemented and no action is needed on this.
2. For the issue 2 above, decrease the complexity of the UI codeI by sending "real" `system.process.cgroup.memory.mem.limit.bytes`. This means, if no memory limit is set for the POD, the agent won't send the "magic" `9223372036854771712L` value, instead, it'll send `system.memory.total` in `system.process.cgroup.memory.mem.limit.bytes`. This means the max memory that a pod can take is the same as the overall system memory. We also need to update the agent spec to describe this behaviour.
## Spec Issue
- [ ] https://github.com/elastic/apm/issues/815
## Agent Issues
- [ ] https://github.com/elastic/apm-agent-java/issues/3213
- [ ] https://github.com/elastic/apm-agent-dotnet/issues/2125
- [ ] https://github.com/elastic/apm-agent-nodejs/issues/3452
- [ ] https://github.com/elastic/apm-agent-python/issues/1866
- [ ] https://github.com/elastic/apm-agent-go/issues/1479
- [ ] https://github.com/elastic/apm-agent-php/issues/1012
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.