getsentry / getsentry/sentry-java
Handle delayed OpenTelemetry span finish calls
- Ngôn ngữ chính
- Kotlin
- Star
- 1.4k
- Fork
- 478
- Merge trung bình
- 2 ngày 23 giờ
- Pull request đã merge (30 ngày)
- 67
Mô tả
## Problem
If a child OpenTelemetry span is finished after its local root span (which Sentry turns into a transaction) and arrives at `SentrySpanExporter` in a later exporter batch than the root span, the child becomes orphaned and is never sent to Sentry.
The root transaction has already been captured by that point, so the late child cannot be appended to it. Today the late child stays in `SentrySpanExporter`'s temporary `finishedSpans` storage as an unset/orphaned span and is eventually dropped after the pending-span timeout.
This is observable with async producer spans, e.g. Kafka:
- HTTP transaction `GET /kafka/produce` finishes and is exported first.
- Kafka producer span `queue.publish` may finish later when the Kafka send callback/broker ack completes.
- If both spans are exported in the same OTel batch, Sentry groups them and the test passes.
- If the HTTP root is exported in an earlier batch, `queue.publish` arrives too late, remains orphaned, and is dropped.
So this is currently flaky because it depends on how OpenTelemetry batches ended spans.
## Current behavior
`SentrySpanExporter` only sends completed root nodes:
- local root span -> Sentry transaction
- child spans under that root -> embedded Sentry spans
- child whose parent is missing -> retained as pending/unset
- pending/unset span older than timeout -> dropped
That means a late `queue.publish` with `parent_span_id` pointing to an already-sent HTTP transaction is not sent at all.
## Expected behavior
Late-finished local child spans should not be silently dropped. Ideally they should still be represented in Sentry, either by giving them a chance to be grouped into the original transaction or by sending them through a model that supports independent span delivery.
## Potential fixes
1. **Wait for span streaming / spans v2**
- Architectural fix: send spans independently instead of embedding all children into a transaction event.
- A child span finishing after the root could still be sent with trace/span/parent metadata.
- This avoids the append-to-already-sent-transaction limitation entirely, but depends on span streaming support for this path.
2. **Debounce root transaction export in `SentrySpanExporter`**
- When a completed local root arrives, hold it for a short grace window before creating/sending the Sentry transaction.
- If late children arrive during that window, group them into the transaction.
- This is a pragmatic improvement and can reduce flakes, but it is timing-based. Children that finish after the grace window can still be missed.
- `flush()` and `shutdown()` would need to drain pending roots immediately so short-lived processes do not lose buffered transactions.
3. **Send orphaned late spans as their own transaction**
- Keep a bounded cache of sent span ids.
- If a late span's parent id is already known as sent, promote the late span to its own transaction and mark it, e.g. `sentry.parent_span_already_sent=true`.
- This avoids silent data loss, similar to the Sentry JavaScript OTel exporter behavior.
- It does not preserve the original shape (`queue.publish` embedded in `GET /kafka/produce`), but it is better than dropping the span.
## E2E test snippet
This assertion is useful but currently flaky because it depends on whether OTel exports the HTTP root and Kafka producer span in the same batch. Keeping the snippet here makes it easy to restore once this behavior is fixed.
```kotlin
package io.sentry.systemtest
import io.sentry.systemtest.util.TestHelper
import kotlin.test.Test
import kotlin.test.assertEquals
import org.junit.Before
class KafkaOtelCoexistenceSystemTest {
lateinit var testHelper: TestHelper
@Before
fun setup() {
testHelper = TestHelper("http://localhost:8080")
testHelper.reset()
}
@Test
fun `Sentry Kafka integration is suppressed when OTel is active`() {
val restClient = testHelper.restClient
restClient.produceKafkaMessage("otel-coexistence-test")
assertEquals(200, restClient.lastKnownStatusCode)
testHelper.ensureTransactionReceived { transaction, _ ->
transaction.transaction == "GET /kafka/produce" &&
transaction.sdk?.integrationSet?.contains("SpringKafka") != true &&
transaction.spans.any { span ->
span.op == "queue.publish" &&
span.origin == "auto.opentelemetry" &&
span.data?.get("messaging.system") == "kafka"
}
}
testHelper.ensureTransactionReceived { transaction, _ ->
transaction.contexts.trace?.operation == "queue.process" &&
transaction.contexts.trace?.origin == "auto.opentelemetry" &&
transaction.contexts.trace?.data?.get("messaging.system") == "kafka" &&
transaction.sdk?.integrationSet?.contains("SpringKafka") != true
}
}
}
```
## Related context
The Kafka producer span can finish after the HTTP request span because Kafka sends are asynchronous. The HTTP handler can return before broker acknowledgement, while OTel Kafka instrumentation ends the producer span from the producer callback.
The failure has been seen in the Spring Boot Jakarta OpenTelemetry system test:
```bash
python3 test/system-test-runner.py test \
--module "sentry-samples-spring-boot-jakarta-opentelemetry" \
--agent true \
--auto-init "false" \
--build "true"
```
Hướng dẫn đóng góp
Đánh giá
Issue này chưa được đánh giá.