High CPU usage (300%+) caused by `keepAliveLoop` goroutine leak and nil pointer panic in `handleWebsocket`
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 16.6k
- Forks
- 1.6k
- PR merge metrics
- No merged PRs in 30d
Description
Description
When running chisel server in a Docker container, CPU usage spikes to 300%+ after client sessions disconnect. The process consumes excessive CPU even with no active tunnels, and eventually becomes unresponsive.
Environment
- Chisel server: v1.11.8
- Chisel client: v1.11.7 (version mismatch observed)
- Running in Docker container
- Reverse tunnel enabled (
-R) - Reverse proxies:
R:2333=>22
Observed Behavior
Top output:
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
3630019 root 20 0 1270408 16980 7048 S 344.0 0.1 8:15 bin
Goroutine stack traces show:
- Multiple
keepAliveLoopgoroutines stuck in[runnable]state, spinning onssh.(*mux).SendRequest - Nil pointer dereference panic in
handleWebsocketatserver_handler.go:94
Logs show:
- Clients rapidly reconnecting with exponential backoff (sessions #4 through #14 in ~7 minutes)
- Version mismatch warnings on every connection
Root Cause Analysis
Bug 1: keepAliveLoop goroutine leak (High)
share/tunnel/tunnel.go:178-193 — The keepAliveLoop function has no context.Context parameter and can only exit when sshConn.SendRequest returns an error:
func (t *Tunnel) keepAliveLoop(sshConn ssh.Conn) {
for {
time.Sleep(t.Config.KeepAlive) // no context awareness
_, b, err := sshConn.SendRequest("ping", true, nil)
if err != nil {
break
}
if len(b) > 0 && !bytes.Equal(b, []byte("pong")) {
t.Debugf("strange ping response")
break
}
}
//close ssh connection on abnormal ping
sshConn.Close()
}
When an SSH connection dies:
- The goroutine may be sleeping in
time.Sleepand won't detect the failure until the nextSendRequestcall (delay up toKeepAliveinterval) SendRequeston a closed mux returns immediately without blocking — the goroutine enters a rapid cycle of:sleep → wake → SendRequest(immediate error) → break → Close → exit- Multiple such goroutines from disconnected sessions remain in
[runnable]state simultaneously, causing the Go scheduler to consume excessive CPU managing transitions
With rapid client reconnect cycles (observed in logs), new keepAliveLoop goroutines are spawned on each BindSSH call before the previous ones have exited, causing goroutine accumulation.
Bug 2: Nil pointer dereference panic (Critical)
server/server_handler.go:94 — When the SSH connection closes before the client sends a "config" request, the reqs channel is closed. Reading from a closed channel returns the zero value (nil for *ssh.Request):
select {
case r = <-reqs: // r = nil when channel is closed
case <-time.After(10 * time.Second):
...
}
if r.Type != "config" { // nil pointer dereference → panic!
This panic kills the handleWebsocket handler, leaving associated goroutines (including keepAliveLoop) orphaned with no parent context to clean them up, further contributing to the goroutine leak.
Suggested Fixes
| Priority | Fix | Location |
|---|---|---|
| Critical | Add nil check: if r == nil { return } after receiving from reqs channel |
server_handler.go:84-88 |
| High | Pass context.Context to keepAliveLoop, replace time.Sleep with context-aware select + time.After |
tunnel.go:178-193 |
| High | Ensure session disconnect explicitly cancels all spawned goroutines | server_handler.go, tunnel.go |
Reproduction
- Run chisel server with reverse tunnel:
chisel server -R --reverse - Connect a client with version mismatch (1.11.7 client → 1.11.8 server)
- Disconnect the client (network interruption or manual stop)
- Observe CPU usage climb as
keepAliveLoopgoroutines accumulate - Reconnect repeatedly to accelerate the issue
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with share/tunnel/tunnel.go:178-193 and server/server_handler.go:84-94, then reproduce the disconnect and reconnect sequence described in the issue while observing CPU and goroutine behavior. Verify that closed-session handling stops the keep-alive work and avoids the nil request panic; done means repeated disconnects no longer accumulate runnable goroutines or cause the handler to panic.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Clearly specified
- Newbie friendliness
- 55/100