[otel_apm_service_map] PRODUCER/CONSUMER spans are not processed — messaging services missing from service map

Description

The otel_apm_service_map processor (v2) does not process PRODUCER and CONSUMER span kinds, which means services that communicate via messaging systems (e.g., RabbitMQ, Kafka) are completely absent from the generated service map.

The legacy service_map processor (v1) does include these services correctly.

Context

We have two NestJS microservices instrumented with OpenTelemetry auto-instrumentation:

• server — REST API, produces messages to RabbitMQ (PRODUCER spans)
• moe-dataset-builder — Async worker, consumes messages from RabbitMQ (CONSUMER spans) and makes DB calls (CLIENT spans)

The trace correlation works perfectly — PRODUCER spans from server have child CONSUMER spans from moe-dataset-builder with matching traceId and parentSpanId. Both services are correctly instrumented and both emit spans to Data Prepper.

Current Behavior

With otel_apm_service_map processor (v2):

• server appears in the service map ✅ (has SERVER + CLIENT spans)
• moe-dataset-builder is completely absent ❌ (has only CLIENT + CONSUMER spans, no SERVER spans)
• The PRODUCER → CONSUMER relationship between the two services is not represented in the service map
• The otel-v2-apm-service-map index contains 0 entries for moe-dataset-builder

With service_map processor (v1):

• Both services appear in the service map ✅
• moe-dataset-builder has 6 entries in the otel-v1-apm-service-map index
• The relationship between the two services is correctly represented

Span Distribution for moe-dataset-builder

SpanKind	Count
CLIENT	74
INTERNAL	10
CONSUMER	2
SERVER	0

Since moe-dataset-builder is a message consumer (not an HTTP server), it legitimately has zero SERVER spans. This is standard for async workers in a microservices architecture.

Root Cause

Looking at the processor README, the Phase 2 algorithm explicitly only processes CLIENT and SERVER spans:

Phase 2 - "CLIENT-Primary" Algorithm

• For each CLIENT span → find relationship to SERVER spans
• For each SERVER span → BFS to find CLIENT descendants

There is no handling for PRODUCER or CONSUMER span kinds at all. This means any service that only communicates via messaging (without receiving HTTP requests) will never appear in the service map.

Expected Behavior

The otel_apm_service_map processor should handle PRODUCER → CONSUMER relationships the same way it handles CLIENT → SERVER relationships:

1. A PRODUCER span should create an edge to the service that has the child CONSUMER span
2. A service with only CONSUMER + CLIENT spans (no SERVER spans) should still appear as a node in the service map
3. The messaging relationship should be represented in the service map, ideally with a distinct "component": "messaging" or similar marker

This would be consistent with:

• The behavior of the legacy service_map processor (which already handles this correctly)
• The OpenTelemetry specification which defines PRODUCER/CONSUMER as first-class span kinds for asynchronous messaging
• Real-world microservice architectures where async messaging (RabbitMQ, Kafka, SQS, etc.) is a fundamental communication pattern

Environment

• Data Prepper: 2.15.0
• OpenSearch: 3.6.0
• OpenTelemetry SDK: Node.js auto-instrumentation 0.71.0
• Messaging system: RabbitMQ (amqplib instrumentation)

Workaround

We currently run both the v1 service_map and v2 otel_apm_service_map processors in parallel pipelines. The v1 index correctly includes the messaging services, but the APM UI in OpenSearch Dashboards only queries the v2 index (otel-v2-apm-service-map), so the messaging services remain invisible in the APM interface.

Related

• Processor source: data-prepper-plugins/otel-apm-service-map-processor
• Legacy processor: data-prepper-plugins/service-map-stateful

Start with the README and source under data-prepper-plugins/otel-apm-service-map-processor, focusing on the Phase 2 CLIENT/SERVER algorithm described in the issue. Compare its handling with data-prepper-plugins/service-map-stateful and trace how PRODUCER and CONSUMER spans are represented. Done means messaging-only services appear as nodes and PRODUCER-to-CONSUMER relationships are present in the v2 service-map output.