opensearch-project / opensearch-project/data-prepper

otel_trace_source drops the whole OTLP request when a span has duplicate attribute keys

Open
#7,151 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

untriaged
Dominant language
Java
Stars
374
Forks
354
Avg merge
3d 18h
Merged PRs (30d)
8

Description

Data Prepper 2.16.0, sink OpenSearch 3.7.0, otel_trace_source over gRPC.

When one span repeats an attribute key, the whole ExportTraceServiceRequest is dropped,
not only that span.

WARN org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceGrpcService -
Failed to parse request with error 'Duplicate key span.attributes.compute@expression
(attempted merging values esql://routine/#MyCompute.Main and
 esql://routine/#MyCompute.Main)'. Request body: resource_spans { ... }

Both merged values are the same, so even an exact duplicate is fatal.

In our traffic every request carried 8 spans, 4 with the duplicated keys and 4 valid root
spans. All 8 were lost and documentsSuccess does not move.

The producer is IBM App Connect Enterprise 13.0.8 with openTelemetryScope: extended. On
Compute node spans it emits four attributes twice, each time with identical value:

spans {
  name: "MyApp.MyFlow.logMessageEvent"
  attributes { key: "compute.expression" value { string_value: "esql://routine/#MyCompute.Main" } }
  attributes { key: "compute.mode"       value { string_value: "message" } }
  attributes { key: "nodeName"           value { string_value: "logMessageEvent" } }
  attributes { key: "nodeType"           value { string_value: "ComputeNode" } }
  attributes { key: "compute.expression" value { string_value: "esql://routine/#MyCompute.Main" } }
  attributes { key: "compute.mode"       value { string_value: "message" } }
  attributes { key: "nodeName"           value { string_value: "logMessageEvent" } }
  attributes { key: "nodeType"           value { string_value: "ComputeNode" } }
}

Pipeline is a plain one, otel_trace_source (port 21890, ssl false, unframed_requests true)
-> otel_traces -> opensearch sink with index_type: trace-analytics-raw.

Cause looks to be in OTelProtoOpensearchCodec$OTelProtoDecoder, which builds the attribute
maps with the two argument Collectors.toMap(keyMapper, valueMapper). That form has no merge
function and throws IllegalStateException on a duplicate key. OTelTraceGrpcService catches
it per request, so everything in the batch is discarded. Same pattern is in
getSpanAttributes, getResourceAttributes, getLinkAttributes and getEventAttributes,
and OTelProtoStandardCodec looks the same.

Expected: a duplicate key should not kill the request. A merge function (a, b) -> b would be
enough, it is lossless when the values are equal. If merging is not wanted, then drop only the
offending span and expose a counter for it.

I am aware the producer is not strictly spec compliant, attribute keys should be unique. But
OTLP protobuf allows a repeated KeyValue, so a receiver will meet this in the wild.
Elasticsearch 9.5.4 native OTLP endpoint accepts the same traffic with 0 errors.

Workarounds we found: set the producer to openTelemetryScope: standard, which removes the
Compute spans (verified, 0 rejected requests), or put an OpenTelemetry Collector in front only
to deduplicate the attributes.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in OTelProtoOpensearchCodec$OTelProtoDecoder, inspecting getSpanAttributes, getResourceAttributes, getLinkAttributes, and getEventAttributes, then compare OTelProtoStandardCodec. Run the relevant codec and otel_trace_source tests, or add focused coverage for repeated attributes. Done means a request containing duplicate keys no longer causes the whole batch to be discarded, while valid spans are processed.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, java
Domain
backend, observability
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
72/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.