awslabs / awslabs/aws-embedded-metrics-node
Unable to set service type, service name and log stream name
- Dominant language
- TypeScript
- Stars
- 263
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
**Problems**
* The variables AWS_EMF_SERVICE_TYPE and AWS_EMF_SERVICE_NAME are not included in the logged metric
* The log stream in AWS_EMF_LOG_STREAM_NAME is ignored
**Config**
* Running on ECS with Fargate (version 1.3.0)
* The agent is in a sidecar
* Using aws-embedded-metrics@2.0.1, Node 12
These are the container definitions:
```
ContainerDefinitions:
# [...]
- Name: app
Image: redacted
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref ContainerLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: app
Environment:
- Name: AWS_EMF_SERVICE_NAME
Value: !Sub ${AWS::StackName}-app
- Name: AWS_EMF_SERVICE_TYPE
Value: "NodeJS-API"
- Name: AWS_EMF_LOG_GROUP_NAME
Value: !Ref ContainerLogGroup
- Name: AWS_EMF_LOG_STREAM_NAME
Value: metrics
- Name: AWS_EMF_NAMESPACE
Value: !Sub ${AWS::StackName}
- Name: AWS_EMF_ENABLE_DEBUG_LOGGING
Value: true
- Name: agent
Image: amazon/cloudwatch-agent:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref ContainerLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: agent
Secrets:
- Name: CW_CONFIG_CONTENT
ValueFrom: !Ref CloudWatchAgentConfigArn
```
This is the agent config:
```
{
"logs": {
"metrics_collected": {
"emf": {}
}
}
}
```
This is what the agent sidecar logs:
```
// Log stream: agent/agent/227b1b1f66744e318edb2d5e9bb57e2d
2020/11/09 16:31:03 I! 2020/11/09 16:31:03 E! ec2metadata is not available
--
2020/11/09 16:31:03 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
I! Detected the instance is ECS
2020/11/09 16:31:03 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directory2020/11/09 16:31:03 unable to scan config dir /etc/cwagentconfig with error: lstat /etc/cwagentconfig: no such file or directory
2020/11/09 16:31:03 Reading json config from from environment variable CW_CONFIG_CONTENT.
Valid Json input schema.
I! detect region from ecs
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
2020/11/09 16:31:03 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020-11-09T16:31:03Z I! Starting AmazonCloudWatchAgent 1.247346.0
2020-11-09T16:31:03Z I! Loaded inputs: socket_listener socket_listener
2020-11-09T16:31:03Z I! Loaded aggregators:
2020-11-09T16:31:03Z I! Loaded processors:
2020-11-09T16:31:03Z I! Loaded outputs: cloudwatchlogs
2020-11-09T16:31:03Z I! Tags enabled:
2020-11-09T16:31:03Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"", Flush Interval:1s
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on udp://[::]:25888
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on tcp://[::]:25888
2020-11-09T16:31:03Z I! [logagent] starting
2020-11-09T16:31:03Z I! [logagent] found plugin cloudwatchlogs is a log backend
```
This is what the app container logs:
```
// Log stream: app/app/227b1b1f66744e318edb2d5e9bb57e2d
Received default dimensions {
LogGroup: 'redactedLogGroupName',
ServiceName: 'my-service-name',
ServiceType: 'NodeJS-API'
}
Sending {} events to socket. 1
opening connection with socket in state: closed
TcpClient connected. { host: '0.0.0.0', port: 25888, protocol: 'tcp:' }
Write succeeded
```
**However**, the metric gets logged to another log stream (not the one in AWS_EMF_LOG_STREAM_NAME) and it does not include the AWS_EMF_SERVICE_TYPE or AWS_EMF_SERVICE_NAME.
```
// Log stream: arn_aws_ecs_eu-north-1_redactedAccountId_task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d
{
"Endpoint": "POST /authentication/v5/login/email",
"Method": "POST",
"Path": "/authentication/v5/login/email",
"StatusCode": 200,
"IP": "redactedIp",
"UserAgent": "Amazon CloudFront",
"containerId": "ip-redactedIp.eu-north-1.compute.internal",
"createdAt": "2020-11-09T16:31:04.591709491Z",
"startedAt": "2020-11-09T16:31:05.226439Z",
"image": "redactedAccountId.dkr.ecr.eu-north-1.amazonaws.com/redactedImage",
"cluster": "arn:aws:ecs:eu-north-1:redactedAccountId:cluster/redactedClusterName",
"taskArn": "arn:aws:ecs:eu-north-1:redactedAccountId:task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d",
"_aws": {
"Timestamp": 1604996779353,
"LogGroupName": "redactedLogGroupName",
"CloudWatchMetrics": [
{
"Dimensions": [
[
"Endpoint"
]
],
"Metrics": [
{
"Name": "Latency",
"Unit": "Milliseconds"
},
{
"Name": "Success",
"Unit": "Count"
}
],
"Namespace": "redactedNamespace"
}
]
},
"Latency": 1415.994263,
"Success": 1
}
```
Contributor guide
Research direction
Start by reproducing the ECS Fargate setup with aws-embedded-metrics@2.0.1, the app and agent container definitions, and the CW_CONFIG_CONTENT configuration shown in the issue. Compare the app and agent logs with the resulting CloudWatch log stream; done means the configured service metadata appears and metrics use AWS_EMF_LOG_STREAM_NAME as expected.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, node.js, typescript
- Domain
- backend, cloud, observability-sre
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100