etcd-io / etcd-io/etcd

Add Distributed Tracing using OpenTelemetry

Open
#12,460 22 comments 13 reactions 0 assignees View on GitHub
priority/important-soon stage/tracked
Dominant language
Go
Stars
52.3k
Forks
10.5k
Avg merge
2d 21h
Merged PRs (30d)
43

Description

Prior issues/PRs related to distributed tracing: https://github.com/etcd-io/etcd/issues/11166, https://github.com/etcd-io/etcd/issues/9242, https://github.com/etcd-io/etcd/pull/5425.

In the kubernetes instrumentation sig, we are adding working to add Distributed Tracing support using OpenTelemetry to the kube-apiserver: [kubernetes/enhancements/keps/sig-instrumentation/647-apiserver-tracing](https://github.com/kubernetes/enhancements/tree/master/keps/sig-instrumentation/647-apiserver-tracing). We plan to have Alpha (disabled by default) support for this feature in kubernetes v1.22. Progress can be tracked in https://github.com/kubernetes/enhancements/issues/647. This will generate spans for incoming and outgoing requests, including requests to etcd. It will also propagate the [w3c trace context](https://www.w3.org/TR/trace-context/) and [w3c baggage](https://github.com/w3c/baggage) along with outgoing requests.

This presents an opportunity to make Etcd easier to debug when using kubernetes. Since distributed tracing provides a unified view of a single request across multiple components, a trace including the API Server and Etcd would show both the API call to the APIServer and the resulting Etcd call, providing better context on the call to Etcd. Based on https://github.com/etcd-io/etcd/pull/5425#issuecomment-226627966, there is already interest in distributed tracing, so I won't elaborate further.

Previous attempts at distributed tracing were [looking into](https://github.com/etcd-io/etcd/pull/5425#issuecomment-360867285) OpenTracing, OpenCensus, and jaeger formats and libraries. Thankfully, these have since unified behind the OpenTelemetry project, which makes a potential evaluation considerably easier. OpenTelemetry will be GA soon, so now seems like the right time to open a discussion around adding distributed tracing to Etcd.

ccing people who have been involved with or interested in this topic.
@logicalhan @tedsuo @ehashman @brancz @bg451 @bhs @gyuho @jingyih @serathius

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.