envoyproxy / envoyproxy/nighthawk
nighthawk_client can hang at startup: unconditional fork() of a multithreaded process
- Dominant language
- C++
- Stars
- 414
- Forks
- 95
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 11
Description
*Title*: nighthawk_client can hang at startup: unconditional fork() of a multithreaded process
*Description*:
`ProcessImpl::run()` always goes through `EncapsulationSubProcessRunner`, which `fork()`s a child even when no `--tunnel-uri` is configured. In that case the child only posts a process-shared semaphore and `_exit()`s while the parent blocks in `sem_wait()`. The parent is already multithreaded at that point (at least the `SignalHandler` thread; in `-c opt` builds also the allocator's state), so the child can deadlock on a lock held by a thread that does not exist in it before reaching `sem_post()`. The client then hangs indefinitely without sending a request. Expected: no fork unless a tunnel is actually requested.
*Reproduction steps*:
Timing dependent. Observed on a 1-worker `nighthawk_client --open-loop --rps 4000 --duration 110 ...` (no tunnel) started in a tight loop of runs against a gRPC server on arm64; roughly 1 in 15 runs hung.
*Logs*:
The last lines before the hang (the fork happens right after them):
```
[I] Starting 1 threads / event loops. Time limit: 110 seconds.
[I] Global targets: 8 connections and 4000 calls per second.
```
then nothing for 38 minutes.
*Call Stack*:
Not a crash. Process state from /proc: parent pid with two threads, main in `__futex_wait` (utime 6 ticks) and one in `anon_pipe_read` (the signal handler thread); a child process with the same argv, single thread, utime 0, never ran.
Contributor guide
Research direction
Start at ProcessImpl::run() and EncapsulationSubProcessRunner, then reproduce with the shown 1-worker nighthawk_client command without --tunnel-uri. Done means startup completes and requests are sent without forking when no tunnel is configured, while tunnel-enabled behavior remains covered by the relevant existing tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- cli
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100