Trace context propagation using non-gRPC headers

Currently, gRPC uses the `grpc-trace-bin` header for context propagation across process boundaries. This works perfectly for an environment in which all services are using gRPC for service-to-service communication. In the case of a more "polyglot transport protocol" environment, preserving the trace context becomes a little more involved.

To give some more specific context / motivation for the problem at hand, we have inbound HTTP requests that are subsequently proxied by an "API gateway"-like service over gRPC to respective backends. All other downstream traffic is gRPC. We're running this in a "mesh" setup, with Envoy running as a proxy alongside each container.

Envoy is configured (via Isito) to look for B3 headers, and will correctly do the context propagation, _for HTTP_. It doesn't know about `grpc-trace-bin` and thus can't participate in these traces. Instead it will emit its own B3-flavored trace for the particular hop, and we end up with incomplete traces - one set for the client / server gRPC spans, and another for just the proxy hops.

I'd like to propose / seek feedback on the idea of making the wire transport for tracing configurable / pluggable. I'm coming at this from the perspective that gRPC's trace propagation is tightly coupled to a gRPC-specfic format and it doesn't really make sense (at least to me) to teach an HTTP server / proxy how to handle gRPC's internal format.

Maybe I'm missing some background. I did a quick look in the issues for this repo but didn't see anything obvious. Looks like something similar has been brought up in the Go community, via census-instrumentation/opencensus-go#666, and census-instrumentation/opencensus-specs#136 (closed out).

If this is better for a list, happy to post there too, jlmk where to ask.

cc: @adriancole

More detail / proof of concept for a Java service:

Basic idea of the change - Open Census has the concept of Binary and TextFormat "setters" and "getters" to do the propagation. We replaced the `grpc-trace-bin` propagation header with the full set of B3 headers (no reason this couldn't be the single B3 header).

I've proven this out internally with some light forking of the code to make a new tracing module that does the B3 propagation. The following code is an example, isn't production ready, etc. etc.

```diff
diff --git a/core/src/main/java/io/grpc/internal/CensusTracingModule.java b/core/src/main/java/io/grpc/internal/CensusTracingModule.java
index b30c02d6a..f0a4dacdd 100644
--- a/core/src/main/java/io/grpc/internal/CensusTracingModule.java
+++ b/core/src/main/java/io/grpc/internal/CensusTracingModule.java
@@ -28,6 +28,7 @@ import io.grpc.Context;
import io.grpc.ForwardingClientCall.SimpleForwardingClientCall;
import io.grpc.ForwardingClientCallListener.SimpleForwardingClientCallListener;
import io.grpc.Metadata;
+import io.grpc.Metadata.Key;
import io.grpc.MethodDescriptor;
import io.grpc.ServerStreamTracer;
import io.grpc.StreamTracer;
@@ -39,7 +40,12 @@ import io.opencensus.trace.Span;
import io.opencensus.trace.SpanContext;
import io.opencensus.trace.Status;
import io.opencensus.trace.Tracer;
+import io.opencensus.trace.Tracing;
import io.opencensus.trace.propagation.BinaryFormat;
+import io.opencensus.trace.propagation.SpanContextParseException;
+import io.opencensus.trace.propagation.TextFormat;
+import io.opencensus.trace.propagation.TextFormat.Getter;
+import io.opencensus.trace.propagation.TextFormat.Setter;
import io.opencensus.trace.unsafe.ContextUtils;
import java.util.concurrent.atomic.AtomicIntegerFieldUpdater;
import java.util.logging.Level;
@@ -59,6 +65,9 @@ import javax.annotation.Nullable;
*/
final class CensusTracingModule {
private static final Logger logger = Logger.getLogger(CensusTracingModule.class.getName());
+ private static final TextFormat B3_FORMAT = Tracing.getPropagationComponent().getB3Format();
+ private static final Getter METADATA_GETTER = new B3Getter();
+ private static final Setter METADATA_SETTER = new B3Setter();

@Nullable private static final AtomicIntegerFieldUpdater callEndedUpdater;

@@ -87,8 +96,6 @@ final class CensusTracingModule {
}

private final Tracer censusTracer;
- @VisibleForTesting
- final Metadata.Key tracingHeader;
private final TracingClientInterceptor clientInterceptor = new TracingClientInterceptor();
private final ServerTracerFactory serverTracerFactory = new ServerTracerFactory();

@@ -96,23 +103,6 @@ final class CensusTracingModule {
Tracer censusTracer, final BinaryFormat censusPropagationBinaryFormat) {
this.censusTracer = checkNotNull(censusTracer, "censusTracer");
checkNotNull(censusPropagationBinaryFormat, "censusPropagationBinaryFormat");
- this.tracingHeader =
- Metadata.Key.of("grpc-trace-bin", new Metadata.BinaryMarshaller() {
- @Override
- public byte[] toBytes(SpanContext context) {
- return censusPropagationBinaryFormat.toByteArray(context);
- }
-
- @Override
- public SpanContext parseBytes(byte[] serialized) {
- try {
- return censusPropagationBinaryFormat.fromByteArray(serialized);
- } catch (Exception e) {
- logger.log(Level.FINE, "Failed to parse tracing header", e);
- return SpanContext.INVALID;
- }
- }
- });
}

/**
@@ -245,8 +235,7 @@ final class CensusTracingModule {
public ClientStreamTracer newClientStreamTracer(
ClientStreamTracer.StreamInfo info, Metadata headers) {
if (span != BlankSpan.INSTANCE) {
- headers.discardAll(tracingHeader);
- headers.put(tracingHeader, span.getContext());
+ B3_FORMAT.inject(span.getContext(), headers, METADATA_SETTER);
}
return new ClientTracer(span);
}
@@ -365,7 +354,13 @@ final class CensusTracingModule {
@SuppressWarnings("ReferenceEquality")
@Override
public ServerStreamTracer newServerStreamTracer(String fullMethodName, Metadata headers) {
- SpanContext remoteSpan = headers.get(tracingHeader);
+ SpanContext remoteSpan = null;
+ try {
+ remoteSpan = B3_FORMAT.extract(headers, METADATA_GETTER);
+ } catch (SpanContextParseException e) {
+ // FIXME: handle this
+ }
+
if (remoteSpan == SpanContext.INVALID) {
remoteSpan = null;
}
@@ -419,4 +414,20 @@ final class CensusTracingModule {
return prefix + "." + fullMethodName.replace('/', '.');
}

+ private static class B3Getter extends Getter {
+
+ @Nullable
+ @Override
+ public String get(Metadata carrier, String key) {
+ return carrier.get(Key.of(key, Metadata.ASCII_STRING_MARSHALLER));
+ }
+ }
+
+ private static class B3Setter extends Setter {
+
+ @Override
+ public void put(Metadata carrier, String key, String value) {
+ carrier.put(Key.of(key, Metadata.ASCII_STRING_MARSHALLER), value);
+ }
+ }
}
```

We're not actually _forking_ gRPC for this, rather we're making a new tracing module that has the above changes, which we then install into client / server pipelines:

```java
// client
CensusB3TracingModule module = new CensusB3TracingModule(Tracing.getTracer());
Channel wrappedChannel = ClientInterceptors.intercept(channel, module.getClientInterceptor());

// server
CensusB3TracingModule censusB3TracingModule = new CensusB3TracingModule(Tracing.getTracer());
NettyServerBuilder.forPort(50051)
.addStreamTracerFactory(censusB3TracingModule.getServerTracerFactory())
.addService(...);
```

We then have to disable the out of the box-gRPC-tracing so that we're only using B3 for propagation:

```java
// client
InternalNettyChannelBuilder.setTracingEnabled(channelBuilder, false);

// server
InternalNettyServerBuilder.setTracingEnabled(serverBuilder, false);
```

With these code changes, we get full end-to-end tracing with all clients, servers and sidecar Envoys adding their spans.