open-telemetry / open-telemetry/opentelemetry-java-instrumentation

Otel Java Auto Instrumentation is setting incorrect container.id resource attribute when application is deployed in AWS ECS Fargate Cluster.

Open
#7,775 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Java
Stars
2.6k
Forks
1.2k
Avg merge
2d 16h
Merged PRs (30d)
214

Description

Describe the bug
Recently ContainerID implementation of containers running in ECS Fargate Cluster has changed. Right now, the ECS Fargate Container uses the containerID format as "32bitAlphanumeric-10bitNumeric" format.

i.e. 8e7e67f77c0849bf80421aebbfbfb045-892424363

Please see the example of one such java application container in ECS Console below:
image

When this application is auto-instrumented with Otel Java Agent, the value of the container.id resource attribute is being set as only the last 10bitNumeric string (i.e. 892424363)

Please see the screenshot from Jaeger below showing the same.
image

This is creating significant problem when later on when we are trying to correlate the application with the actual container instance.

Steps to reproduce
a. Create an ECS Fargate Cluster in AWS.
b. Deploy a basic java web application in the ECS Fargate Cluster and instrument with OpenTelemetry JavaAgent.
c. From AWS ECS Console, verify the container runtime id for the task deployed.
c. Apply load on the application and either export the Spans to an Otel Collector or Jaeger.
d. Look at the traces generated in the Jaeger and find the value of the conntainer.id resource attribute and compare the value with the container runtime id value from AWS ECS Console. You'll see the difference mentioned above.

If you want to reuse my application, I am attaching the jar file and the Dockerfile. In the Dockerfile, I have mentioned all the environment variable that I am using for Otel Instrumention (commented in the file).

What did you expect to see?
The value of the container.id resource attribute set by Otel should match the value of the actual containerID that you see in the AWS ECS Task Console, i.e. 8e7e67f77c0849bf80421aebbfbfb045-892424363.

What did you see instead?
The value of the container.id resource attribute set by Otel is the last 10 digits of the actual containerID that you see in the AWS ECS Task Console, i.e. 892424363.

What version are you using?
Latest Version Of OpenTelemetry javaagent.jar (1.22.1)

Environment
Compiler: Oracle JDK 1.8.0_281
OS: Mac OS Ventura 13.2 (22D49)
Runtime (if different from JDK above): OpenJDK 1.8.0_332-b09
OS (if different from OS compiled on): AWS ECS Fargate Cluster (Linux 4.14.301-224.520.amzn2.x86_64)

TestCase.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the attached TestCase.zip and Dockerfile, reproduce the behavior using the OpenTelemetry Java agent in an AWS ECS Fargate deployment, and trace how the container.id resource attribute is obtained. Compare the emitted value with the full ECS container runtime ID; done means the resource attribute preserves the complete ID shown in the ECS Task Console.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
cloud, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.