aws / aws/containers-roadmap

Allow environment variables in awslogs-group

Open
#360 17 comments 24 reactions 0 assignees View on GitHub
ECS Proposed
Dominant language
Shell
Stars
5.4k
Forks
334
PR merge metrics
No merged PRs in 30d

Description

At my company, we use multicontainer ElasticBeanstalk tiers for all of our environments. Currently we stream logs to SumoLogic via their agent; but, for our needs, CloudWatch logs would be sufficient and much cheaper. Each of our environments defines an environment variable "ENVIRONMENT", with values like "develop", "qa", etc. I found that recent ECS agents support streaming logs to CloudWatch natively via the awslogs driver. I attempted to do something like this:

{
"name": "tasks",
"image": "876270261134.dkr.ecr.us-west-2.amazonaws.com/armada/tasks:{{ BUILD_TAG }}",
"readonlyRootFilesystem": true,
"essential": true,
"memory": 3072,
"workingDirectory": "/armada/app",
"command": [
"./start.sh"
],
"portMappings": [
{
"hostPort": 80,
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/armada/$ENVIRONMENT/tasks.log",
"awslogs-region": "us-west-2"
}
}

Some background: when our CI system produces a build, it instantiates the Dockerrun.aws.json from a template defined in our source repository. `BUILD_TAG` is `${branch}-${build_number}-${commit_hash_prefix}`. The CI system produces ZIP archives for the web and worker tiers, uploads them to S3, and registers them as application versions which can be reused against any of the other environments via a Slack bot (`@bula deploy develop develop-38-abcd1234`).

I don't want to hardcode `"awslogs-group": "/armada/tasks.log"`, because then all environments' logs would be grouped the same. When I attempted this, however, I got an error `CannotStartContainerError: API error (500): Failed to initialize logging driver: InvalidParameterException: 1 validation error detected: Value '\''/armada/$ENVIRONMENT/tasks.log'\'' at '\''logGroupName'\'' failed to satisfy constraint: Member must satisfy regular ex`.

The only workaround I can think of would be to alter our CI system to not register application versions at build time, but just ZIP archives with Dockerrun.aws.json still in Jinja2 template form; and, in tandem, alter the deployment bot to download that, extract it, instantiate the Dockerrun.aws.json template with the target environment, and register a new application version, and update the target environment with it. That seems needlessly convoluted (and, frankly, ugly) to me. If there's some other, cleaner way to accomplish this, I'm open to any suggestions. But IMHO, it'd be quite nice to use environment variables in task definitions, to keep them easily reusable.

Contributor guide

Open the contributing guide

Research direction

Start with the Dockerrun.aws.json example and the ECS awslogs driver behavior described in the issue. Define the supported environment-variable behavior for awslogs-group and verify that reusable task definitions can produce environment-specific log groups without CI-side templating.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws
Domain
cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.