aws / aws/containers-roadmap

[ECS] [request] running an ecs task should allow awslogs-stream-prefix being overridden

Open
#1,618 2 comments 18 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

### Community Note

* Please vote on this issue by adding a 👍 [reaction](https://blog.github.com/2016-03-10-add-reactions-to-pull-requests-issues-and-comments/) to the original issue to help the community and maintainers prioritize this request
* Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
* If you are interested in working on this issue or have submitted a pull request, please leave a comment

**Tell us about your request**

When using code like this python script using `boto3.client('ecs').run_task()`, I should be able to specify a `awslogs-stream-prefix` for the awslogs driver within the container overrides. This way, I can clearly see which log stream corresponds to which fargate ecs task. Here's an example of using `run_task()`:

```
result = ecs_client.run_task(
cluster=clusterName,
taskDefinition=taskDefinitionName,
launchType='FARGATE',
networkConfiguration={
'awsvpcConfiguration': {
'subnets': [
...
]
}
},
overrides={
'containerOverrides': [
{
'name': taskDefinitionName,
// let me specify a log driver override with options like
'command': [
'/bin/sh',
'-c',
f'make deploy_qa env=qa-{build_id} args="{arg_string}"',
]
}
]
}
)
```

**Which service(s) is this request for?**

ECS

**Tell us about the problem you're trying to solve. What are you trying to do, and why is it hard?**

Right now, when I look at my log streams for this ecs task definition (see screenshot below), I have to find logs either by searching or using the timestamp. However, each log stream within my log group does correspond 1:1 with a build_id - but it's too hard to find which log stream is which, because we have so many of these running in parallel.

When creating a task definition I can specify a stream prefix BUT when running the task, I can specify all sorts of overrides, except for the stream prefix. Here are the docs on specifying the stream prefix during task definition creation. https://docs.aws.amazon.com/AmazonECS/latest/developerguide/using_awslogs.html

**Are you currently working around this issue?**

yes, currently I attempt to find the right stream by the timestamp which isn't too hard but it's not straightforward.

**Additional context**

I've opened a support task and they told me it's not possible and to request a community issue. Case id 9419954151

**Attachments**

Here's a screenshot of my log group. Each stream corresponds to a separate build_id, yet I have no way of getting the build_id to appear in this listing.

![image](https://user-images.githubusercontent.com/15964/148094776-81129630-f5f8-47f4-9139-5a87d37be635.png)

Contributor guide

Open the contributing guide

Research direction

Start with the ECS run_task API and the linked awslogs documentation, focusing on the containerOverrides schema and how awslogs-stream-prefix is defined in task definitions. Define what an override should accept and verify that the requested build-specific prefix is reflected in the resulting Fargate log stream.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, infrastructure
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.