envoyproxy / envoyproxy/envoy

Go dynamic module deadlocks integration-test server teardown on macOS

Open
#46,905 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
28.9k
Forks
5.6k
Avg merge
1d 22h
Merged PRs (30d)
430

Description

**Corrected from the original report.** I first described this as the request hanging. It isn't: the request succeeds and the test body passes. The hang is in test *teardown*, and it looks macOS-specific.

With a Go dynamic module loaded, `~IntegrationTestServer` never returns. `sample` on the stuck process shows the main thread parked here:

```
~DynamicModulesIntegrationTest -> ~HttpIntegrationTest -> ~BaseIntegrationTest
-> ~IntegrationTestServerImpl -> IntegrationTestServer::~IntegrationTestServer()
-> Thread::PosixThread::join() -> _pthread_join -> __ulock_wait
```

and the thread it is waiting on parked here:

```
thread_start -> _pthread_start -> _pthread_exit -> _pthread_tsd_cleanup
-> _sigtramp -> ??? (in )
```

So the server thread is exiting, macOS runs its pthread thread-specific-data destructors, one of them enters the loaded Go module, and it never comes back. `pthread_join` then blocks forever.

Repro: any test with a `direct_response` route and a Go module filter, which is what I was adding for #46901. It also reproduces with the in-tree `passthrough` filter, so it has nothing to do with what the filter does — only that a Go module is loaded. `rust`, `rust_static` and `cpp` on the identical test finish in under a second. Go is the only one of the three SDKs whose runtime installs pthread TSD destructors, which fits.

Two things suggest this is macOS-only and test-only rather than a production problem. A Go dynamic module in a long-running Linux Envoy container handles the same `direct_response` route fine over both HTTP/1.1 and HTTP/2 — but that process never tears a server thread down with the module loaded, so it does not exercise this path. And CI is Linux, so this likely never shows up there.

Not sure what the right fix is. Possibly the server thread should not be joined while a dynamic module holding thread-local state is still loaded, or the module should be unloaded first, or this is a Go runtime interaction that Envoy can only work around.

Repro harness: https://github.com/derekargueta/envoy — the test is in the #46902 branch, which skips the `go` param on Apple platforms and cites this issue.

main @ 8ea3b35553, macOS arm64.

Contributor guide

Open the contributing guide

Research direction

Use the linked repro harness and the #46902 test, starting with DynamicModulesIntegrationTest teardown and the IntegrationTestServerImpl to Thread::PosixThread::join stack. Compare the Go dynamic-module case with rust, rust_static, and cpp, then verify that the direct_response test request still succeeds and teardown returns on macOS without hanging.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp, go, macos
Domain
operating-systems, testing
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.