aws / aws/containers-roadmap

.net core application running on fargate with memory issues

Open
#1,927 1 comment 0 reactions 0 assignees View on GitHub
Docs ECS Fargate
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

We are running a dotnet application in fargate via terraform where we specify cpu and memory in the aws_ecs_task_definition resource. The service has just 1 task e.g.

```
resource "aws_ecs_task_definition" "test" {
....
cpu = 256
memory = 512
....
```
From the documentation this is required for Fargate. You can also specify cpu and memory in the container_definitions, but the
[documentation](https://docs.aws.amazon.com/AmazonECS/latest/APIReference/API_ContainerDefinition.html#:%7E:text=This%20field%20is%20optional%20for%20tasks%20using%20the%20Fargate%20launch%20type) states that the field is optional, and as we are already setting values at the task level we did not set them here.

We have observed that our memory was growing after the tasks started, depending on application, sometimes quite fast and others over a period of time.

So we starting thinking we had a memory leak and went to profile using the dotnet-monitor tool as a sidecar. As part of introducing the sidecar we set cpu and memory values for our dotnet application at the container_definitions level.

After we done this, we have observed that our memory in our applications is behaving much better. From dotnet monitor traces we are seeing that when we set memory at the container_definitions level:

Working Set is much smaller
Gen 0/1/2 GC Count is above 1(GC occurring early)
GC 0/1/2 Size is less
GC Committed Bytes is smaller
So to summarize when we do not set memory at container_definitions level, memory continues to grow and no GC occurring until we are almost running out of memory.

When we set memory at container_definitions level, GC occurring regularly and memory not spiking up.

So we have a solution, but do not understand why this is the case. Would like to know why it is so and should the documentation be updated to specify that these values should be set at the container_definitions level also?

Contributor guide

Open the contributing guide

Research direction

Start with the AWS ECS task definition and ContainerDefinition documentation linked in the issue, then review the shown Terraform aws_ecs_task_definition and container_definitions settings alongside the dotnet-monitor observations. Determine whether the documentation accurately explains the different memory settings and clarify the required guidance if it does not.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, terraform
Domain
cloud, infrastructure
Issue type
Documentation
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.