awslabs / awslabs/aws-embedded-metrics-java

Race in EnvironmentProvider.resolveEnvironment() yields non-deterministic ECSEnvironment.getLogGroupName() under FireLens

Open
#168 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
48
Forks
39
PR merge metrics
No merged PRs in 30d

Description

When multiple threads with their own MetricsLoggers are spawned there is a race to assign the static EnvironmentProvider.cachedEnvironment via [EnvironmentProvider.resolveEnvironment()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/EnvironmentProvider.java#L43) because multiple threads can slip past the `if (cachedEnvironment != null)` guard before the associated CompletableFutures have resolved. This can lead to concurrent efforts to resolve [ECSEnvironment via ECSEnvironment.probe()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L57), and non-deterministic behaviour from [ECSEnvironment.getLogGroupName](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L101).

Part of setting up ECSEnvironment is calling [ECSEnvironment.checkAndSetFluentHost()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L138). In that method both a per-instance property `fluentBitEndpoint` and a (shared singleton) Configuration config.agentEndpoint property are set. The [guard for this assignment](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L140) checks that config.agentEndpoint is not already set, and if it is then neither the local `fluentBitEndpoint` nor `config.agentEndpoint` are touched. When two or more threads are simultaneously resolving ECSEnvironment on a FluentBit host and no `config.agentEndpoint` already exists, then the first thread will pass the `!config.getAgentEndpoint().isPresent()` guard and set the local `fluentBitEndpoint`. Subsequent threads will detect that the shared `config.agentEndpoint` is present and will NOT set the local `fluentBitEndpoint`.

When [ECSEnvironment.getLogGroupName()](https://github.com/awslabs/aws-embedded-metrics-java/blob/36538f307c7a1500422a97a00ee9684f9c5a74d2/src/main/java/software/amazon/cloudwatchlogs/emf/environment/ECSEnvironment.java#L101) is called, if `this.fluentBitEndpoint` is set then an empty string log group name is returned, otherwise a full log group name. Taken together this means that when multiple ECSEnvironments are being resolved simultaneously on a FluentBit host, the first will produce an empty string log group name, while any subsequent ones will produce a full log group name. This makes the race to assign in EnvironmentProvider consequential, since any of these ECSEnvironments could theoretically end up as the authoritative reference saved in cachedEnvironment. This can (and did!) lead to inconsistent startup behaviour where sometimes LogGroupName is empty and sometimes it is populated.

**Impact**: on ECS + FireLens, at cold start under concurrent first Metric emissions, a task can non-deterministically cache an ECSEnvironment whose getLogGroupName() is "", which in our case drops the LogGroup default dimension for the lifetime of the JVM. I observed this; identical tasks intermittently emit metrics with vs. without the LogGroup dimension, silently breaking dashboards/alarms keyed on it.

Contributor guide

Open the contributing guide

Research direction

Start by reading EnvironmentProvider.resolveEnvironment(), then trace ECSEnvironment.probe(), checkAndSetFluentHost(), and getLogGroupName(). Reproduce concurrent first resolution on an ECS FireLens host and inspect how cachedEnvironment and agentEndpoint are assigned. Done means concurrent startup deterministically preserves the expected LogGroup behavior instead of intermittently dropping the dimension.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
backend, observability
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.