opensearch-project / opensearch-project/data-prepper
otel_trace_source drops the whole OTLP request when a span has duplicate attribute keys
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 374
- Forks
- 354
- Avg merge
- 3d 18h
- Merged PRs (30d)
- 8
Description
Data Prepper 2.16.0, sink OpenSearch 3.7.0, otel_trace_source over gRPC.
When one span repeats an attribute key, the whole ExportTraceServiceRequest is dropped,
not only that span.
WARN org.opensearch.dataprepper.plugins.source.oteltrace.OTelTraceGrpcService -
Failed to parse request with error 'Duplicate key span.attributes.compute@expression
(attempted merging values esql://routine/#MyCompute.Main and
esql://routine/#MyCompute.Main)'. Request body: resource_spans { ... }
Both merged values are the same, so even an exact duplicate is fatal.
In our traffic every request carried 8 spans, 4 with the duplicated keys and 4 valid root
spans. All 8 were lost and documentsSuccess does not move.
The producer is IBM App Connect Enterprise 13.0.8 with openTelemetryScope: extended. On
Compute node spans it emits four attributes twice, each time with identical value:
spans {
name: "MyApp.MyFlow.logMessageEvent"
attributes { key: "compute.expression" value { string_value: "esql://routine/#MyCompute.Main" } }
attributes { key: "compute.mode" value { string_value: "message" } }
attributes { key: "nodeName" value { string_value: "logMessageEvent" } }
attributes { key: "nodeType" value { string_value: "ComputeNode" } }
attributes { key: "compute.expression" value { string_value: "esql://routine/#MyCompute.Main" } }
attributes { key: "compute.mode" value { string_value: "message" } }
attributes { key: "nodeName" value { string_value: "logMessageEvent" } }
attributes { key: "nodeType" value { string_value: "ComputeNode" } }
}
Pipeline is a plain one, otel_trace_source (port 21890, ssl false, unframed_requests true)
-> otel_traces -> opensearch sink with index_type: trace-analytics-raw.
Cause looks to be in OTelProtoOpensearchCodec$OTelProtoDecoder, which builds the attribute
maps with the two argument Collectors.toMap(keyMapper, valueMapper). That form has no merge
function and throws IllegalStateException on a duplicate key. OTelTraceGrpcService catches
it per request, so everything in the batch is discarded. Same pattern is in
getSpanAttributes, getResourceAttributes, getLinkAttributes and getEventAttributes,
and OTelProtoStandardCodec looks the same.
Expected: a duplicate key should not kill the request. A merge function (a, b) -> b would be
enough, it is lossless when the values are equal. If merging is not wanted, then drop only the
offending span and expose a counter for it.
I am aware the producer is not strictly spec compliant, attribute keys should be unique. But
OTLP protobuf allows a repeated KeyValue, so a receiver will meet this in the wild.
Elasticsearch 9.5.4 native OTLP endpoint accepts the same traffic with 0 errors.
Workarounds we found: set the producer to openTelemetryScope: standard, which removes the
Compute spans (verified, 0 rejected requests), or put an OpenTelemetry Collector in front only
to deduplicate the attributes.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in OTelProtoOpensearchCodec$OTelProtoDecoder, inspecting getSpanAttributes, getResourceAttributes, getLinkAttributes, and getEventAttributes, then compare OTelProtoStandardCodec. Run the relevant codec and otel_trace_source tests, or add focused coverage for repeated attributes. Done means a request containing duplicate keys no longer causes the whole batch to be discarded, while valid spans are processed.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, java
- Domain
- backend, observability
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 72/100