envoyproxy / envoyproxy/nighthawk

nighthawk_client can hang at startup: unconditional fork() of a multithreaded process

Open
#1,609 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
414
Forks
95
Avg merge
2d 9h
Merged PRs (30d)
11

Description

*Title*: nighthawk_client can hang at startup: unconditional fork() of a multithreaded process

*Description*:
`ProcessImpl::run()` always goes through `EncapsulationSubProcessRunner`, which `fork()`s a child even when no `--tunnel-uri` is configured. In that case the child only posts a process-shared semaphore and `_exit()`s while the parent blocks in `sem_wait()`. The parent is already multithreaded at that point (at least the `SignalHandler` thread; in `-c opt` builds also the allocator's state), so the child can deadlock on a lock held by a thread that does not exist in it before reaching `sem_post()`. The client then hangs indefinitely without sending a request. Expected: no fork unless a tunnel is actually requested.

*Reproduction steps*:
Timing dependent. Observed on a 1-worker `nighthawk_client --open-loop --rps 4000 --duration 110 ...` (no tunnel) started in a tight loop of runs against a gRPC server on arm64; roughly 1 in 15 runs hung.

*Logs*:
The last lines before the hang (the fork happens right after them):
```
[I] Starting 1 threads / event loops. Time limit: 110 seconds.
[I] Global targets: 8 connections and 4000 calls per second.
```
then nothing for 38 minutes.

*Call Stack*:
Not a crash. Process state from /proc: parent pid with two threads, main in `__futex_wait` (utime 6 ticks) and one in `anon_pipe_read` (the signal handler thread); a child process with the same argv, single thread, utime 0, never ran.

Contributor guide

Open the contributing guide

Research direction

Start at ProcessImpl::run() and EncapsulationSubProcessRunner, then reproduce with the shown 1-worker nighthawk_client command without --tunnel-uri. Done means startup completes and requests are sent without forking when no tunnel is configured, while tunnel-enabled behavior remains covered by the relevant existing tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
cli
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.