aws / aws/amazon-cloudwatch-agent
Question/Feature request: how can I prepend an environment name to configured log groups at runtime
- Dominant language
- Go
- Stars
- 550
- Forks
- 271
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 13
Description
**Scenario**
We have implemented a 'golden image' pipeline using AWS ImageBuilder that bakes in our base cloudwatch agent configuration (core logs files and system metrics),
e.g.
```
...
"logs": {
"log_stream_name": "${aws:InstanceId}",
"force_flush_interval" : 60,
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/syslog",
"log_group_name": "syslog",
"timestamp_format": "%b %d %H:%M:%S"
},
...
```
This ends up with a massive log group with streams of every instance we're running in the region. We want to be able to separate those out into say '/production/syslog' and '/staging/syslog'...
**Work Around**
Our current work-around is to run `sed` over the base config included in the golden image early in the boot order...
e.g. `sed -E 's/"log_group_name": "(.*)"/"log_group_name": "\/ENV_NAME\/TIER\/\1"/g' /path/to/amazon-cloudwatch-agent.json`
```
"logs": {
"log_stream_name": "${aws:InstanceId}",
"force_flush_interval" : 60,
"logs_collected": {
"files": {
"collect_list": [
{
"file_path": "/var/log/syslog",
"log_group_name": "/ENV_NAME/TIER/syslog",
"timestamp_format": "%b %d %H:%M:%S"
},
{
"file_path": "/var/log/auth",
"log_group_name": "/ENV_NAME/TIER/auth",
"timestamp_format": "%b %d %H:%M:%S"
},
{
"file_path": "/var/log/audit/audit.log",
"log_group_name": "/ENV_NAME/TIER/audit"
},
{
"file_path": "/var/log/aide/aide.log",
"log_group_name": "/ENV_NAME/TIER/aide"
}
]
}
}
```
**Desired Behaviour**
It would be great if we could just declare arbitrary environment variables that are picked up by and agent and substituted in at runtime. e.g. `"log_group_name": "/${env.ENV_NAME}/${env.TIER}/aide"`
Contributor guide
Research direction
No source files or tests are identified in the issue. Start by tracing how the agent loads and expands its configuration, then determine where runtime environment-variable substitution for log_group_name would be handled. Done means configured environment values produce the intended separated log-group names.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, go
- Domain
- cloud, observability-sre
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100