[FEATURE] Enhance OpenTelemetry tracing diagnostics / 增强 OpenTelemetry Trace 诊断能力
- Dominant language
- Go
- Stars
- 5k
- Forks
- 1k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 31
Description
## Background / 背景
Dubbo-go already supports OpenTelemetry tracing through client/server filters, propagators, samplers, and multiple exporters such as stdout, Jaeger, Zipkin, OTLP HTTP, and OTLP gRPC. The current tracing path is usable, but production troubleshooting needs richer span names, attributes, events, and error information.
Dubbo-go 目前已经通过 client/server filter、propagator、sampler 以及 stdout、Jaeger、Zipkin、OTLP HTTP、OTLP gRPC 等 exporter 支持 OpenTelemetry tracing。当前链路已经可用,但生产排障还需要更丰富的 span name、attributes、events 和错误信息。
## Goals / 目标
Enhance OpenTelemetry tracing so Dubbo-go traces are not only connected end-to-end, but also useful for diagnosing latency, routing, retry, timeout, serialization, registry, and downstream failure problems.
增强 OpenTelemetry tracing 能力,让 Dubbo-go trace 不只是端到端连通,还能真正用于诊断延迟、路由、重试、超时、序列化、注册中心以及下游调用失败等问题。
## Proposed Scope / 建议范围
### 1. Standardize span names and semantic attributes / 标准化 span name 和语义属性
Review and standardize client/server span naming. Candidate format:
梳理并标准化 client/server span 命名。候选格式:
- Consumer span: `dubbo.consumer /`
- Provider span: `dubbo.provider /`
- Consumer span:`dubbo.consumer /`
- Provider span:`dubbo.provider /`
Review and enrich span attributes, including:
梳理并补充 span attributes,包括:
- `rpc.system=apache_dubbo`
- `rpc.service`
- `rpc.method`
- `rpc.grpc.status_code` or Dubbo-specific equivalent where applicable
- `network.peer.address` / `server.address` where available
- Dubbo-specific attributes such as side, protocol, group, version, and error type
- `rpc.system=apache_dubbo`
- `rpc.service`
- `rpc.method`
- `rpc.grpc.status_code` 或适用场景下的 Dubbo 等价字段
- 可获取时记录 `network.peer.address` / `server.address`
- Dubbo 特有属性,例如 side、protocol、group、version、error type
The final attribute names should align with OpenTelemetry semantic conventions where available and use a stable `dubbo.*` namespace for Dubbo-specific data.
最终属性命名应尽量对齐 OpenTelemetry semantic conventions;Dubbo 特有信息应使用稳定的 `dubbo.*` 命名空间。
### 2. Add diagnostic span events / 增加诊断型 span events
Add span events for key runtime decisions and failure points where the information is available:
在信息可获取的前提下,为关键运行时决策和失败点增加 span events:
- Retry attempts and final retry result.
- Load-balance selection result.
- Router or tag-route decision.
- Timeout and cancellation.
- Rate limit or circuit-breaking rejection.
- Serialization/codec failure.
- Registry lookup or provider-empty cases.
- 重试次数及最终重试结果。
- 负载均衡选择结果。
- 路由或 tag-route 决策。
- 超时和取消。
- 限流或熔断拒绝。
- 序列化/codec 失败。
- 注册中心 lookup 或 provider 为空场景。
These events should be low-noise and should not expose sensitive request payloads.
这些 events 应保持低噪音,并且不能暴露敏感请求 payload。
### 3. Improve error recording / 改进错误记录
Current spans set error status from `result.Error()`. This can be made more useful by:
当前 span 主要基于 `result.Error()` 设置 error status,可以进一步增强:
- Recording a stable error type/category, reusing the RPC metrics error taxonomy where possible.
- Adding error code attributes for Triple/gRPC and Dubbo protocol errors.
- Avoiding lossy conversion of structured errors into plain strings where a typed error is available.
- Ensuring span status, span events, and logs can be correlated.
- 记录稳定的 error type/category,并尽可能复用 RPC metrics 的错误分类体系。
- 为 Triple/gRPC 和 Dubbo protocol 错误增加 error code attributes。
- 当存在 typed error 时,避免把结构化错误降级为普通字符串。
- 确保 span status、span events 和 logs 可以相互关联。
### 4. Clarify propagation behavior / 明确传播行为
Document and test propagation behavior for:
文档化并测试以下传播行为:
- W3C trace context.
- B3 propagation.
- Dubbo attachments used as carriers.
- Interaction with business attachments and baggage.
- Consumer -> provider propagation across Triple and Dubbo protocol paths where supported.
- W3C trace context。
- B3 propagation。
- 作为 carrier 使用的 Dubbo attachments。
- 与业务 attachments 和 baggage 的交互。
- 在支持的场景下覆盖 Consumer -> Provider 跨 Triple 和 Dubbo protocol 的传播。
### 5. Improve trace/log/metric correlation / 改进 trace、log、metric 关联
Coordinate tracing with existing observability features:
将 tracing 与现有可观测能力联动:
- Correlate with `CtxLogger` trace fields (`trace_id`, `span_id`, `trace_flags`).
- Reuse metrics error classification where applicable.
- Document how a user moves from a Grafana panel to a trace and then to related logs.
- 与 `CtxLogger` trace fields(`trace_id`、`span_id`、`trace_flags`)关联。
- 在适用场景下复用 metrics 错误分类。
- 文档说明用户如何从 Grafana 面板跳转到 trace,再定位相关日志。
## Acceptance Criteria / 验收标准
- Client and server spans have documented, stable names and attributes.
- Span events exist for at least retry/timeout/rejection/codec or equivalent diagnostic points where supported.
- Error spans include stable category/code attributes in addition to human-readable messages.
- Propagation behavior is covered by tests for supported propagators.
- Documentation and samples explain how tracing connects with metrics and context-aware logging.
- Client 和 server spans 具备文档化、稳定的名称和属性。
- 在支持的场景下,至少为 retry/timeout/rejection/codec 或等价诊断点增加 span events。
- Error spans 除可读错误信息外,还包含稳定的 category/code attributes。
- 支持的 propagator 传播行为有测试覆盖。
- 文档和示例说明 tracing 如何与 metrics、context-aware logging 关联。
## Related Context / 相关上下文
- Existing OpenTelemetry implementation: `otel/trace/*`, `filter/otel/trace/*`, `config/otel_config.go`
- Logger trace correlation: https://github.com/apache/dubbo-go/pull/3195
- Logger sample: https://github.com/apache/dubbo-go-samples/pull/1030
- 现有 OpenTelemetry 实现:`otel/trace/*`、`filter/otel/trace/*`、`config/otel_config.go`
- 日志 trace 关联能力:https://github.com/apache/dubbo-go/pull/3195
- 日志示例:https://github.com/apache/dubbo-go-samples/pull/1030
Contributor guide
Assessment
This issue has not been assessed yet.