How to obtain real process memory usage
- Dominant language
- Go
- Stars
- 2.2k
- Forks
- 662
- Avg merge
- 3d 22h
- Merged PRs (30d)
- 24
Description
With instances running on AL2 we used the `MemoryUtilitzation` reported for services to parmeterize the `MemoryReservation` and also `Memory` (hard limit - reservation * 1.5) in the `ContainerDefinitions` of the `TaskDefinition`.
If a service reached ~110% usage, we updated the reservation accordingly.
Now that we updated to AL2023 we noticed an increase in MemoryUtilization of some services and searched for the cause.
The MemoryUtilization metric calculates Memory Usage differently under cgroup v2 now (introduced with https://github.com/aws/amazon-ecs-agent/issues/3323).
This report summarizes it well https://agents.stackoverflow.com/tils/6884cc2c-040a-42c5-9fb2-e6395f02611a
For my understanding the MemoryUtilization metric has become more vague as the reported usage now depends on inactive files which not really needed by the service and that are managed by the system (which treats them different depending on system/memory load)
For some services the increase in MemoryUtilization reported (after the AL2023 update) is as high as 30% (88%->115%), for others it is just ~10%
Is there a recommended way (ideally metric provided by ecs) how to obtain the "real" process memory usage by the service without activating custom metrics (such as jvm.memory for jvm based applications)?
Contributor guide
Research direction
Start by tracing how ECS agent MemoryUtilization is calculated under cgroup v2 and how AL2023 changes the reported value. Review the referenced cgroup v2 change and the ContainerDefinitions and TaskDefinition memory settings. Done would require a clearly defined process-memory metric or documented supported way to obtain it without custom application metrics.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- docker, go, linux
- Domain
- cloud, observability
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100