Allow environment variables in awslogs-group
- Dominant language
- Shell
- Stars
- 5.4k
- Forks
- 334
- PR merge metrics
- No merged PRs in 30d
Description
At my company, we use multicontainer ElasticBeanstalk tiers for all of our environments. Currently we stream logs to SumoLogic via their agent; but, for our needs, CloudWatch logs would be sufficient and much cheaper. Each of our environments defines an environment variable "ENVIRONMENT", with values like "develop", "qa", etc. I found that recent ECS agents support streaming logs to CloudWatch natively via the awslogs driver. I attempted to do something like this:
{
"name": "tasks",
"image": "876270261134.dkr.ecr.us-west-2.amazonaws.com/armada/tasks:{{ BUILD_TAG }}",
"readonlyRootFilesystem": true,
"essential": true,
"memory": 3072,
"workingDirectory": "/armada/app",
"command": [
"./start.sh"
],
"portMappings": [
{
"hostPort": 80,
"containerPort": 80
}
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/armada/$ENVIRONMENT/tasks.log",
"awslogs-region": "us-west-2"
}
}
Some background: when our CI system produces a build, it instantiates the Dockerrun.aws.json from a template defined in our source repository. `BUILD_TAG` is `${branch}-${build_number}-${commit_hash_prefix}`. The CI system produces ZIP archives for the web and worker tiers, uploads them to S3, and registers them as application versions which can be reused against any of the other environments via a Slack bot (`@bula deploy develop develop-38-abcd1234`).
I don't want to hardcode `"awslogs-group": "/armada/tasks.log"`, because then all environments' logs would be grouped the same. When I attempted this, however, I got an error `CannotStartContainerError: API error (500): Failed to initialize logging driver: InvalidParameterException: 1 validation error detected: Value '\''/armada/$ENVIRONMENT/tasks.log'\'' at '\''logGroupName'\'' failed to satisfy constraint: Member must satisfy regular ex`.
The only workaround I can think of would be to alter our CI system to not register application versions at build time, but just ZIP archives with Dockerrun.aws.json still in Jinja2 template form; and, in tandem, alter the deployment bot to download that, extract it, instantiate the Dockerrun.aws.json template with the target environment, and register a new application version, and update the target environment with it. That seems needlessly convoluted (and, frankly, ugly) to me. If there's some other, cleaner way to accomplish this, I'm open to any suggestions. But IMHO, it'd be quite nice to use environment variables in task definitions, to keep them easily reusable.
Contributor guide
Research direction
Start with the Dockerrun.aws.json example and the ECS awslogs driver behavior described in the issue. Define the supported environment-variable behavior for awslogs-group and verify that reusable task definitions can produce environment-specific log groups without CI-side templating.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws
- Domain
- cloud
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100