awslabs / awslabs/aws-embedded-metrics-java
Race in EnvironmentProvider.resolveEnvironment() yields non-deterministic ECSEnvironment.getLogGroupName() under FireLens
- Dominant language
- Java
- Stars
- 48
- Forks
- 39
- PR merge metrics
- No merged PRs in 30d
Description
When multiple threads with their own MetricsLoggers are spawned there is a race to assign the static EnvironmentProvider.cachedEnvironment via [EnvironmentProvider.resolveEnvironment()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/EnvironmentProvider.java#L43) because multiple threads can slip past the `if (cachedEnvironment != null)` guard before the associated CompletableFutures have resolved. This can lead to concurrent efforts to resolve [ECSEnvironment via ECSEnvironment.probe()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L57), and non-deterministic behaviour from [ECSEnvironment.getLogGroupName](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L101).
Part of setting up ECSEnvironment is calling [ECSEnvironment.checkAndSetFluentHost()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L138). In that method both a per-instance property `fluentBitEndpoint` and a (shared singleton) Configuration config.agentEndpoint property are set. The [guard for this assignment](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L140) checks that config.agentEndpoint is not already set, and if it is then neither the local `fluentBitEndpoint` nor `config.agentEndpoint` are touched. When two or more threads are simultaneously resolving ECSEnvironment on a FluentBit host and no `config.agentEndpoint` already exists, then the first thread will pass the `!config.getAgentEndpoint().isPresent()` guard and set the local `fluentBitEndpoint`. Subsequent threads will detect that the shared `config.agentEndpoint` is present and will NOT set the local `fluentBitEndpoint`.
When [ECSEnvironment.getLogGroupName()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L101) is called, if `this.fluentBitEndpoint` is set then an empty string log group name is returned, otherwise a full log group name. Taken together this means that when multiple ECSEnvironments are being resolved simultaneously on a FluentBit host, the first will produce an empty string log group name, while any subsequent ones will produce a full log group name. This makes the race to assign in EnvironmentProvider consequential, since any of these ECSEnvironments could theoretically end up as the authoritative reference saved in cachedEnvironment. This can (and did!) lead to inconsistent startup behaviour where sometimes LogGroupName is empty and sometimes it is populated.
**Impact**: on ECS + FireLens, at cold start under concurrent first Metric emissions, a task can non-deterministically cache an ECSEnvironment whose getLogGroupName() is "", which in our case drops the LogGroup default dimension for the lifetime of the JVM. I observed this; identical tasks intermittently emit metrics with vs. without the LogGroup dimension, silently breaking dashboards/alarms keyed on it.
Contributor guide
Research direction
Start by reading EnvironmentProvider.resolveEnvironment(), then trace ECSEnvironment.probe(), checkAndSetFluentHost(), and getLogGroupName(). Reproduce concurrent first resolution on an ECS FireLens host and inspect how cachedEnvironment and agentEndpoint are assigned. Done means concurrent startup deterministically preserves the expected LogGroup behavior instead of intermittently dropping the dimension.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100