googleapis / googleapis/google-api-go-client
OpenTelemetry traces misused by transport/http
- Dominant language
- Go
- Stars
- 4.5k
- Forks
- 1.5k
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 32
Description
The `transport/http` package by default [wraps the HTTP transport](https://github.com/googleapis/google-api-go-client/blob/v0.82.0/transport/http/dial.go#L208) with an OpenTelemetry [`ochttp.Transport`](https://pkg.go.dev/go.opencensus.io/plugin/ochttp#Transport). That transport wrapper will record HTTP traces for a subset of all requests if OpenCensus tracing is enabled.
However, it is misconfigured. The `Transport` type has this field, which is unset `google-api-go-client`:
```
// NameFromRequest holds the function to use for generating the span name
// from the information found in the outgoing HTTP Request. By default the
// name equals the URL Path.
FormatSpanName func(*http.Request) string
```
The default is for the transport to name each span after the request URL path. In practice, this means a proliferation of differently named spans. For example, if you use the Google Cloud Storage library, you will end up with a span for every file you access. This is not how spans are supposed to be named - they should be named after a function or API endpoint, perhaps including some small, finite set of user-specified options.
This becomes pathological if you actually record the traces somewhere. For instance, if you have the OpenCensus [`zpages`](https://pkg.go.dev/go.opencensus.io/zpages) debug endpoints enabled (common in production systems), a sample of those traces will be stored in a local store. While old traces for a given span are purged, the spans themselves are never purged, and so a running process will accumulate traces indefinitely for every HTTP request path made by `google-api-go-client`. Loading the OpenCensus `tracez` page in such cases is pretty funny: a production service of mine had accumulated several gigabytes of traces, and its `tracez` endpoint served so much HTML that it crashed my browser.
I think the fix here is to specify a `FormatSpanName` function when setting up the `ochttp.Transport`. I'm not sure how exactly the spans should be named - probably in some service-specific way - but an immediate remedy would be to give all requests the same span name (`google-api-go-client`?).
Contributor guide
Assessment
This issue has not been assessed yet.