awslabs / awslabs/aws-embedded-metrics-node

Unable to set service type, service name and log stream name

Open
#66 5 comments 0 reactions 0 assignees View on GitHub
needs-investigation
Dominant language
TypeScript
Stars
263
Forks
39
PR merge metrics
No merged PRs in 30d

Description

**Problems**

* The variables AWS_EMF_SERVICE_TYPE and AWS_EMF_SERVICE_NAME are not included in the logged metric
* The log stream in AWS_EMF_LOG_STREAM_NAME is ignored

**Config**

* Running on ECS with Fargate (version 1.3.0)
* The agent is in a sidecar
* Using aws-embedded-metrics@2.0.1, Node 12

These are the container definitions:

```
ContainerDefinitions:
# [...]
- Name: app
Image: redacted
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref ContainerLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: app
Environment:
- Name: AWS_EMF_SERVICE_NAME
Value: !Sub ${AWS::StackName}-app
- Name: AWS_EMF_SERVICE_TYPE
Value: "NodeJS-API"
- Name: AWS_EMF_LOG_GROUP_NAME
Value: !Ref ContainerLogGroup
- Name: AWS_EMF_LOG_STREAM_NAME
Value: metrics
- Name: AWS_EMF_NAMESPACE
Value: !Sub ${AWS::StackName}
- Name: AWS_EMF_ENABLE_DEBUG_LOGGING
Value: true
- Name: agent
Image: amazon/cloudwatch-agent:latest
LogConfiguration:
LogDriver: awslogs
Options:
awslogs-group: !Ref ContainerLogGroup
awslogs-region: !Ref AWS::Region
awslogs-stream-prefix: agent
Secrets:
- Name: CW_CONFIG_CONTENT
ValueFrom: !Ref CloudWatchAgentConfigArn
```

This is the agent config:

```
{
"logs": {
"metrics_collected": {
"emf": {}
}
}
}
```

This is what the agent sidecar logs:

```
// Log stream: agent/agent/227b1b1f66744e318edb2d5e9bb57e2d

2020/11/09 16:31:03 I! 2020/11/09 16:31:03 E! ec2metadata is not available
--
2020/11/09 16:31:03 I! attempt to access ECS task metadata to determine whether I'm running in ECS.
I! Detected the instance is ECS
2020/11/09 16:31:03 Reading json config file path: /opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json ...
/opt/aws/amazon-cloudwatch-agent/bin/default_linux_config.json does not exist or cannot read. Skipping it.
Cannot access /etc/cwagentconfig: lstat /etc/cwagentconfig: no such file or directory2020/11/09 16:31:03 unable to scan config dir /etc/cwagentconfig with error: lstat /etc/cwagentconfig: no such file or directory
2020/11/09 16:31:03 Reading json config from from environment variable CW_CONFIG_CONTENT.
Valid Json input schema.
I! detect region from ecs
No csm configuration found.
No metric configuration found.
Configuration validation first phase succeeded
 
2020/11/09 16:31:03 I! Config has been translated into TOML /opt/aws/amazon-cloudwatch-agent/etc/amazon-cloudwatch-agent.toml
2020-11-09T16:31:03Z I! Starting AmazonCloudWatchAgent 1.247346.0
2020-11-09T16:31:03Z I! Loaded inputs: socket_listener socket_listener
2020-11-09T16:31:03Z I! Loaded aggregators:
2020-11-09T16:31:03Z I! Loaded processors:
2020-11-09T16:31:03Z I! Loaded outputs: cloudwatchlogs
2020-11-09T16:31:03Z I! Tags enabled:
2020-11-09T16:31:03Z I! [agent] Config: Interval:1m0s, Quiet:false, Hostname:"", Flush Interval:1s
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on udp://[::]:25888
2020-11-09T16:31:03Z I! [inputs.socket_listener] Listening on tcp://[::]:25888
2020-11-09T16:31:03Z I! [logagent] starting
2020-11-09T16:31:03Z I! [logagent] found plugin cloudwatchlogs is a log backend
```

This is what the app container logs:

```
// Log stream: app/app/227b1b1f66744e318edb2d5e9bb57e2d

Received default dimensions {
LogGroup: 'redactedLogGroupName',
ServiceName: 'my-service-name',
ServiceType: 'NodeJS-API'
}
Sending {} events to socket. 1
opening connection with socket in state: closed
TcpClient connected. { host: '0.0.0.0', port: 25888, protocol: 'tcp:' }
Write succeeded
```

**However**, the metric gets logged to another log stream (not the one in AWS_EMF_LOG_STREAM_NAME) and it does not include the AWS_EMF_SERVICE_TYPE or AWS_EMF_SERVICE_NAME.

```
// Log stream: arn_aws_ecs_eu-north-1_redactedAccountId_task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d

{
"Endpoint": "POST /authentication/v5/login/email",
"Method": "POST",
"Path": "/authentication/v5/login/email",
"StatusCode": 200,
"IP": "redactedIp",
"UserAgent": "Amazon CloudFront",
"containerId": "ip-redactedIp.eu-north-1.compute.internal",
"createdAt": "2020-11-09T16:31:04.591709491Z",
"startedAt": "2020-11-09T16:31:05.226439Z",
"image": "redactedAccountId.dkr.ecr.eu-north-1.amazonaws.com/redactedImage",
"cluster": "arn:aws:ecs:eu-north-1:redactedAccountId:cluster/redactedClusterName",
"taskArn": "arn:aws:ecs:eu-north-1:redactedAccountId:task/redactedClusterName/227b1b1f66744e318edb2d5e9bb57e2d",
"_aws": {
"Timestamp": 1604996779353,
"LogGroupName": "redactedLogGroupName",
"CloudWatchMetrics": [
{
"Dimensions": [
[
"Endpoint"
]
],
"Metrics": [
{
"Name": "Latency",
"Unit": "Milliseconds"
},
{
"Name": "Success",
"Unit": "Count"
}
],
"Namespace": "redactedNamespace"
}
]
},
"Latency": 1415.994263,
"Success": 1
}
```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the ECS Fargate setup with aws-embedded-metrics@2.0.1, the app and agent container definitions, and the CW_CONFIG_CONTENT configuration shown in the issue. Compare the app and agent logs with the resulting CloudWatch log stream; done means the configured service metadata appears and metrics use AWS_EMF_LOG_STREAM_NAME as expected.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, node.js, typescript
Domain
backend, cloud, observability-sre
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.