googleapis / googleapis/google-cloud-go

auth/httptransport: default otelhttp overwrites T4 span error.type

Open
#14,441 0 comments 0 reactions 1 assignee Assigned to @shollyman View on GitHub
priority: p2 type: bug
Dominant language
Go
Stars
4.5k
Forks
1.6k
Avg merge
1d 13h
Merged PRs (30d)
109

Description

## Client
`auth/httptransport` (affecting generated REST clients using T4 tracing)

## Environment
* Go version: `go version go1.25.0 linux/amd64`
* OS: Linux

## Code and Dependencies

```go
package main

import (
"context"
"errors"
"fmt"
"net/http"

"go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
"go.opentelemetry.io/otel"
"go.opentelemetry.io/otel/attribute"
"go.opentelemetry.io/otel/sdk/trace"
"go.opentelemetry.io/otel/sdk/trace/tracetest"
oteltrace "go.opentelemetry.io/otel/trace"
)

// mockTransport always returns context.DeadlineExceeded to simulate a timeout.
type mockTransport struct{}

func (t *mockTransport) RoundTrip(req *http.Request) (*http.Response, error) {
return nil, context.DeadlineExceeded
}

// otelAttributeTransport simulates the inner transport in auth/httptransport.
// It attempts to set a custom error.type attribute.
type otelAttributeTransport struct {
base http.RoundTripper
}

func (t *otelAttributeTransport) RoundTrip(req *http.Request) (*http.Response, error) {
resp, err := t.base.RoundTrip(req)
span := oteltrace.SpanFromContext(req.Context())
if span.IsRecording() {
if err != nil {
var errorType string
switch {
case errors.Is(err, context.DeadlineExceeded):
errorType = "CLIENT_TIMEOUT"
default:
errorType = "UNKNOWN"
}
// Set the custom attribute
span.SetAttributes(
attribute.String("error.type", errorType),
attribute.String("test.source", "otelAttributeTransport"),
)
}
}
return resp, err
}

func main() {
// 1. Setup in-memory exporter to inspect spans
exporter := tracetest.NewInMemoryExporter()
tp := trace.NewTracerProvider(trace.WithSyncer(exporter))
otel.SetTracerProvider(tp)

// 2. Build transport stack mimicking auth/httptransport
mock := &mockTransport{}
inner := &otelAttributeTransport{base: mock}
outer := otelhttp.NewTransport(inner) // otelhttp wraps our custom transport

client := &http.Client{
Transport: outer,
}

// 3. Run request
ctx := context.Background()
req, _ := http.NewRequestWithContext(ctx, "GET", "http://example.com", nil)

_, err := client.Do(req)
if err == nil {
fmt.Println("Expected error, got nil")
return
}

// 4. Inspect spans
spans := exporter.GetSpans()
if len(spans) == 0 {
fmt.Println("No spans recorded!")
return
}

span := spans[0]
fmt.Printf("Span Name: %s\n", span.Name)
fmt.Println("Attributes:")
for _, attr := range span.Attributes {
fmt.Printf(" %s: %v\n", attr.Key, attr.Value.AsInterface())
}
}
```

go.mod

```text
module repro

go 1.25.0

require (
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0
go.opentelemetry.io/otel v1.42.0
go.opentelemetry.io/otel/sdk v1.42.0
go.opentelemetry.io/otel/trace v1.42.0
)
```

## Expected behavior
The `error.type` attribute on the span should be mapped to `"CLIENT_TIMEOUT"` (or `"CLIENT_CANCELLED"`, etc.) as defined in `otelAttributeTransport` error mapping logic.

## Actual behavior
The `error.type` attribute is overwritten with the raw Go error type string (e.g., `"context.deadlineExceededError"`).

Output of reproduction:
```
Span Name: HTTP GET
Attributes:
http.request.method: GET
url.full: http://example.com
server.address: example.com
network.protocol.version: 1.1
error.type: context.deadlineExceededError
test.source: otelAttributeTransport
```

## Additional context
In the `cloud.google.com/go/auth/httptransport` package, the OpenTelemetry instrumentation wraps the transport in a way that causes `otelhttp` to overwrite the custom `error.type` attribute value set by our internal `otelAttributeTransport`.

Specifically, in `auth/httptransport/transport.go`:
```go
trans = &otelAttributeTransport{base: trans}
// ...
return otelhttp.NewTransport(trans, otelOpts...)
```

`otelhttp.NewTransport` (the outer transport) wraps `otelAttributeTransport` (the inner transport).
When a request fails (e.g., due to a context timeout):
1. `otelAttributeTransport` runs first and correctly maps the error (e.g., `context.DeadlineExceeded` -> `"CLIENT_TIMEOUT"`) and sets it on the span.
2. It returns the error to `otelhttp`.
3. `otelhttp` sees the error and, following its default behavior, records the exception and sets `error.type` to the raw Go error type (e.g., `"context.deadlineExceededError"`).
4. This overwrites the specialized value set by `otelAttributeTransport`.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.