googleapis / googleapis/google-api-go-client

OpenTelemetry traces misused by transport/http

Open
#1,573 5 comments 1 reaction 1 assignee Claimed by @shollyman View on GitHub
needs more info priority: p2 type: bug
Dominant language
Go
Stars
4.5k
Forks
1.5k
Avg merge
1d 15h
Merged PRs (30d)
32

Description

The `transport/http` package by default [wraps the HTTP transport](https://github.com/googleapis/google-api-go-client/blob/v0.82.0/transport/http/dial.go#L208) with an OpenTelemetry [`ochttp.Transport`](https://pkg.go.dev/go.opencensus.io/plugin/ochttp#Transport). That transport wrapper will record HTTP traces for a subset of all requests if OpenCensus tracing is enabled.

However, it is misconfigured. The `Transport` type has this field, which is unset `google-api-go-client`:

```
// NameFromRequest holds the function to use for generating the span name
// from the information found in the outgoing HTTP Request. By default the
// name equals the URL Path.
FormatSpanName func(*http.Request) string
```

The default is for the transport to name each span after the request URL path. In practice, this means a proliferation of differently named spans. For example, if you use the Google Cloud Storage library, you will end up with a span for every file you access. This is not how spans are supposed to be named - they should be named after a function or API endpoint, perhaps including some small, finite set of user-specified options.

This becomes pathological if you actually record the traces somewhere. For instance, if you have the OpenCensus [`zpages`](https://pkg.go.dev/go.opencensus.io/zpages) debug endpoints enabled (common in production systems), a sample of those traces will be stored in a local store. While old traces for a given span are purged, the spans themselves are never purged, and so a running process will accumulate traces indefinitely for every HTTP request path made by `google-api-go-client`. Loading the OpenCensus `tracez` page in such cases is pretty funny: a production service of mine had accumulated several gigabytes of traces, and its `tracez` endpoint served so much HTML that it crashed my browser.

I think the fix here is to specify a `FormatSpanName` function when setting up the `ochttp.Transport`. I'm not sure how exactly the spans should be named - probably in some service-specific way - but an immediate remedy would be to give all requests the same span name (`google-api-go-client`?).

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.