runtime: SIGSEGV in buildTypeAssertCache during crypto/tls TLS 1.3 client handshake
- Dominant language
- Go
- Stars
- 139k
- Forks
- 19.4k
- PR merge metrics
- PR metrics pending
Description
### Go version
go1.26.5 linux/amd64 (built with the official golang:1.26.5 Docker image, CGO_ENABLED=1)
### Output of `go env` in your module/workspace:
```shell
GOOS='linux'
GOARCH='amd64'
GOAMD64='v1'
CGO_ENABLED='1'
GOVERSION='go1.26.5'
# The crash occurred in a production container; full 'go env' of the build environment is not available.
# The binary was built inside the official golang:1.26.5 image with default settings.
```
### What did you do?
A long-running agent (running in a container on a Kubernetes node) makes periodic HTTPS requests to the Kubernetes API server via `k8s.io/client-go` v0.34.1 / `net/http`. After ~58.7 hours of uptime, one routine request crashed inside the TLS 1.3 client handshake with a SIGSEGV in `runtime.buildTypeAssertCache`.
This happened once and is not reproducible on demand. The binary does use cgo. I am mostly filing this because the signature looks very similar to #78220 (also a fault on a bad pointer inside `runtime.typeAssert` in the crypto/tls handshake path, also sporadic on a long-running Kubernetes workload), in case another sample helps.
Note `code=0x80` (SI_KERNEL) with fault address 0x0, faulting pc equals the `runtime.buildTypeAssertCache` frame itself (iface.go:544, i.e. the sort in `buildTypeAssertCache.func1` at iface.go:532).
### What did you see happen?
Crashing goroutine, its creator, and the goroutine that initiated the request (application frames below the `controller-runtime` client call omitted). Everything is verbatim from the crash dump.
```
unexpected fault address 0x0
fatal error: fault
[signal SIGSEGV: segmentation violation code=0x80 addr=0x0 pc=0x43a200]
goroutine 167455 gp=0x194eb4ab3860 m=10 mp=0x194eb50d3008 [running]:
runtime.throw({0x1cc1092?, 0x90?})
/usr/local/go/src/runtime/panic.go:1229 +0x48 fp=0x194eb4efd990 sp=0x194eb4efd960 pc=0x4a08c8
runtime.sigpanic()
/usr/local/go/src/runtime/signal_unix.go:945 +0x285 fp=0x194eb4efd9f0 sp=0x194eb4efd990 pc=0x4a2b85
runtime.buildTypeAssertCache.func1(...)
/usr/local/go/src/runtime/iface.go:532
runtime.buildTypeAssertCache(0x1ab1020?, 0x1ab0820, 0x7f4c8661a050)
/usr/local/go/src/runtime/iface.go:544 +0x140 fp=0x194eb4efda38 sp=0x194eb4efd9f0 pc=0x43a200
runtime.typeAssert(0x2e32ea0, 0x1ab0820?)
/usr/local/go/src/runtime/iface.go:497 +0xdd fp=0x194eb4efda78 sp=0x194eb4efda38 pc=0x439ffd
crypto/tls.transcriptMsg({0x1df5dd8, 0x194eb4e832c0}, {0x7f4c8661a000, 0x194eb52f4080})
/usr/local/go/src/crypto/tls/handshake_messages.go:1952 +0x10b fp=0x194eb4efdab0 sp=0x194eb4efda78 pc=0x69a88b
crypto/tls.(*clientHandshakeStateTLS13).handshake(0x194eb4efdc80)
/usr/local/go/src/crypto/tls/handshake_client_tls13.go:119 +0x6b8 fp=0x194eb4efdb70 sp=0x194eb4efdab0 pc=0x67d978
crypto/tls.(*Conn).clientHandshake(0x194eb4a2ea88, {0x1e084a0, 0x194eb4cfc370})
/usr/local/go/src/crypto/tls/handshake_client.go:339 +0x828 fp=0x194eb4efdd88 sp=0x194eb4efdb70 pc=0x6772a8
crypto/tls.(*Conn).clientHandshake-fm({0x1e084a0?, 0x194eb4cfc370?})
:1 +0x2a fp=0x194eb4efddb0 sp=0x194eb4efdd88 pc=0x6b57ea
crypto/tls.(*Conn).handshakeContext(0x194eb4a2ea88, {0x1e084a0, 0x194eb4cfc320})
/usr/local/go/src/crypto/tls/conn.go:1562 +0x329 fp=0x194eb4efdf70 sp=0x194eb4efddb0 pc=0x66e3c9
crypto/tls.(*Conn).HandshakeContext(...)
/usr/local/go/src/crypto/tls/conn.go:1516
net/http.(*persistConn).addTLS.func2()
/usr/local/go/src/net/http/transport.go:1736 +0x6e fp=0x194eb4efdfe0 sp=0x194eb4efdf70 pc=0x7539ce
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1771 +0x1 fp=0x194eb4efdfe8 sp=0x194eb4efdfe0 pc=0x4a8221
created by net/http.(*persistConn).addTLS in goroutine 167454
/usr/local/go/src/net/http/transport.go:1732 +0x309
goroutine 167454 gp=0x194eb50405a0 m=nil [chan receive]:
runtime.gopark(0x194eb4f4b818?, 0x60000004a4b50?, 0x80?, 0xe0?, 0x7f4ccf392a00?)
/usr/local/go/src/runtime/proc.go:462 +0xce fp=0x194eb4f4b7c8 sp=0x194eb4f4b7a8 pc=0x4a09ee
runtime.chanrecv(0x194eb49fd880, 0x194eb4f4b980, 0x1)
/usr/local/go/src/runtime/chan.go:667 +0x4ae fp=0x194eb4f4b840 sp=0x194eb4f4b7c8 pc=0x434aee
runtime.chanrecv1(0x194eb4a145a0?, 0x194eb49b9ec0?)
/usr/local/go/src/runtime/chan.go:509 +0x12 fp=0x194eb4f4b868 sp=0x194eb4f4b840 pc=0x434612
net/http.(*persistConn).addTLS(0x194eb4e83040, {0x1e084a0, 0x194eb4cfc320}, {0x194eb4f95610, 0xb}, 0x194eb4de9000)
/usr/local/go/src/net/http/transport.go:1742 +0x327 fp=0x194eb4f4b9a8 sp=0x194eb4f4b868 pc=0x7536c7
net/http.(*Transport).dialConn(0x194eb4b076c0, {0x1e084a0, 0x194eb4cfc320}, {{}, 0x0, {0x194eb513c0c0, 0x5}, {0x194eb4f95610, 0xf}, 0x0}, ...)
/usr/local/go/src/net/http/transport.go:1825 +0x9fc fp=0x194eb4f4bed0 sp=0x194eb4f4b9a8 pc=0x7544bc
net/http.(*Transport).dialConnFor(0x194eb4b076c0, 0x194eb4a362c0)
/usr/local/go/src/net/http/transport.go:1648 +0xd2 fp=0x194eb4f4bf90 sp=0x194eb4f4bed0 pc=0x752d52
net/http.(*Transport).startDialConnForLocked.func1()
/usr/local/go/src/net/http/transport.go:1629 +0x35 fp=0x194eb4f4bfe0 sp=0x194eb4f4bf90 pc=0x752b95
runtime.goexit({})
/usr/local/go/src/runtime/asm_amd64.s:1771 +0x1 fp=0x194eb4f4bfe8 sp=0x194eb4f4bfe0 pc=0x4a8221
created by net/http.(*Transport).startDialConnForLocked in goroutine 167433
/usr/local/go/src/net/http/transport.go:1628 +0x112
goroutine 167433 gp=0x194eb50e92c0 m=nil [select]:
runtime.gopark(0x194eb4ac27a0?, 0x2?, 0xc8?, 0x25?, 0x194eb4ac26d4?)
/usr/local/go/src/runtime/proc.go:462 +0xce fp=0x194eb4ac2568 sp=0x194eb4ac2548 pc=0x4a09ee
runtime.selectgo(0x194eb4ac27a0, 0x194eb4ac26d0, 0xf?, 0x0, 0x194eb4d67a00?, 0x1)
/usr/local/go/src/runtime/select.go:351 +0xaa5 fp=0x194eb4ac2698 sp=0x194eb4ac2568 pc=0x47e9a5
net/http.(*Transport).getConn(0x194eb4b076c0, 0x194eb4cfc2d0, {{}, 0x0, {0x194eb513c0c0, 0x5}, {0x194eb4f95610, 0xf}, 0x0})
/usr/local/go/src/net/http/transport.go:1552 +0x4b0 fp=0x194eb4ac2870 sp=0x194eb4ac2698 pc=0x752290
net/http.(*Transport).roundTrip(0x194eb4b076c0, 0x194eb4e82dc0)
/usr/local/go/src/net/http/transport.go:693 +0xa7f fp=0x194eb4ac2a50 sp=0x194eb4ac2870 pc=0x74deff
net/http.(*Transport).RoundTrip(0x1b80bc0?, 0x194eb516a450?)
/usr/local/go/src/net/http/roundtrip.go:33 +0x18 fp=0x194eb4ac2a70 sp=0x194eb4ac2a50 pc=0x75dd18
k8s.io/client-go/transport.(*bearerAuthRoundTripper).RoundTrip(0x194eb4eb5d40, 0x194eb4e82b40)
/app/vendor/k8s.io/client-go/transport/round_trippers.go:323 +0x3f2 fp=0x194eb4ac2b90 sp=0x194eb4ac2a70 pc=0xc841f2
k8s.io/client-go/transport.(*userAgentRoundTripper).RoundTrip(0x194eb4cbd560, 0x194eb4e828c0)
/app/vendor/k8s.io/client-go/transport/round_trippers.go:174 +0x337 fp=0x194eb4ac2c80 sp=0x194eb4ac2b90 pc=0xc82f37
net/http.send(0x194eb4e828c0, {0x1dedc00, 0x194eb4cbd560}, {0x194eb4ac2ef8?, 0x4a3506?, 0x0?})
/usr/local/go/src/net/http/client.go:264 +0x64b fp=0x194eb4ac2e70 sp=0x194eb4ac2c80 pc=0x6eeb4b
net/http.(*Client).send(0x194eb4eb5d70, 0x194eb4e828c0, {0x194eb4ac2f28?, 0x4a148f?, 0x0?})
/usr/local/go/src/net/http/client.go:185 +0x258 fp=0x194eb4ac2f08 sp=0x194eb4ac2e70 pc=0x6ee398
net/http.(*Client).do(0x194eb4eb5d70, 0x194eb4e828c0)
/usr/local/go/src/net/http/client.go:733 +0x9d7 fp=0x194eb4ac30f8 sp=0x194eb4ac2f08 pc=0x6f0937
net/http.(*Client).Do(...)
/usr/local/go/src/net/http/client.go:592
k8s.io/client-go/rest.(*Request).request(0x194eb4a3e180, {0x1e08510, 0x194eb49fcc40}, 0x194eb4ac32e0)
/app/vendor/k8s.io/client-go/rest/request.go:1088 +0x3ca fp=0x194eb4ac3208 sp=0x194eb4ac30f8 pc=0xcb6e2a
k8s.io/client-go/rest.(*Request).Do(0x194eb4a3e180, {0x1e08510, 0x194eb49fcc40})
/app/vendor/k8s.io/client-go/rest/request.go:1136 +0x145 fp=0x194eb4ac3320 sp=0x194eb4ac3208 pc=0xcb7765
sigs.k8s.io/controller-runtime/pkg/client.(*typedClient).Get(0x194eb4ec0d80, {0x1e08510, 0x194eb49fcc40}, {{0x0?, 0x194eb4ac34f8?}, {0x194eb49b202a?, 0x10?}}, {0x1e20d78, 0x194eb4cc2008}, {0x194eb49b9cc0, ...})
/app/vendor/sigs.k8s.io/controller-runtime/pkg/client/typed_client.go:174 +0x24f fp=0x194eb4ac3498 sp=0x194eb4ac3320 pc=0x166c44f
sigs.k8s.io/controller-runtime/pkg/client.(*client).Get(0x194eb4ec0d80, {0x1e08510, 0x194eb49fcc40}, {{0x0?, 0x4c4a1a?}, {0x194eb49b202a?, 0x0?}}, {0x1e20d78, 0x194eb4cc2008}, {0x194eb49b9cc0, ...})
/app/vendor/sigs.k8s.io/controller-runtime/pkg/client/client.go:359 +0x2bb fp=0x194eb4ac3600 sp=0x194eb4ac3498 pc=0x166155b
...application frames omitted...
```
### What did you expect to see?
No crash.
Contributor guide
Research direction
Start with runtime/iface.go around buildTypeAssertCache at lines 532-544, then follow the type assertion from crypto/tls/handshake_messages.go:1952 through the TLS 1.3 client handshake. Compare the failure signature with issue #78220 and investigate the long-running, sporadic cgo-enabled workload. Done means identifying the cause of the invalid access and preventing the crash with coverage for the failing path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- compilers, security
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100