jpillora / jpillora/chisel

High CPU usage (300%+) caused by `keepAliveLoop` goroutine leak and nil pointer panic in `handleWebsocket`

Open
#608 3 comments 2 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Go
Stars
16.6k
Forks
1.6k
PR merge metrics
No merged PRs in 30d

Description

Description

When running chisel server in a Docker container, CPU usage spikes to 300%+ after client sessions disconnect. The process consumes excessive CPU even with no active tunnels, and eventually becomes unresponsive.

Environment

  • Chisel server: v1.11.8
  • Chisel client: v1.11.7 (version mismatch observed)
  • Running in Docker container
  • Reverse tunnel enabled (-R)
  • Reverse proxies: R:2333=>22

Observed Behavior

Top output:

PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND
3630019 root      20   0 1270408  16980   7048 S 344.0   0.1      8:15 bin

Goroutine stack traces show:

  • Multiple keepAliveLoop goroutines stuck in [runnable] state, spinning on ssh.(*mux).SendRequest
  • Nil pointer dereference panic in handleWebsocket at server_handler.go:94

Logs show:

  • Clients rapidly reconnecting with exponential backoff (sessions #4 through #14 in ~7 minutes)
  • Version mismatch warnings on every connection

Root Cause Analysis

Bug 1: keepAliveLoop goroutine leak (High)

share/tunnel/tunnel.go:178-193 — The keepAliveLoop function has no context.Context parameter and can only exit when sshConn.SendRequest returns an error:

func (t *Tunnel) keepAliveLoop(sshConn ssh.Conn) {
    for {
        time.Sleep(t.Config.KeepAlive)  // no context awareness
        _, b, err := sshConn.SendRequest("ping", true, nil)
        if err != nil {
            break
        }
        if len(b) > 0 && !bytes.Equal(b, []byte("pong")) {
            t.Debugf("strange ping response")
            break
        }
    }
    //close ssh connection on abnormal ping
    sshConn.Close()
}

When an SSH connection dies:

  1. The goroutine may be sleeping in time.Sleep and won't detect the failure until the next SendRequest call (delay up to KeepAlive interval)
  2. SendRequest on a closed mux returns immediately without blocking — the goroutine enters a rapid cycle of: sleep → wake → SendRequest(immediate error) → break → Close → exit
  3. Multiple such goroutines from disconnected sessions remain in [runnable] state simultaneously, causing the Go scheduler to consume excessive CPU managing transitions

With rapid client reconnect cycles (observed in logs), new keepAliveLoop goroutines are spawned on each BindSSH call before the previous ones have exited, causing goroutine accumulation.

Bug 2: Nil pointer dereference panic (Critical)

server/server_handler.go:94 — When the SSH connection closes before the client sends a "config" request, the reqs channel is closed. Reading from a closed channel returns the zero value (nil for *ssh.Request):

select {
case r = <-reqs:   // r = nil when channel is closed
case <-time.After(10 * time.Second):
    ...
}
if r.Type != "config" {  // nil pointer dereference → panic!

This panic kills the handleWebsocket handler, leaving associated goroutines (including keepAliveLoop) orphaned with no parent context to clean them up, further contributing to the goroutine leak.

Suggested Fixes

Priority Fix Location
Critical Add nil check: if r == nil { return } after receiving from reqs channel server_handler.go:84-88
High Pass context.Context to keepAliveLoop, replace time.Sleep with context-aware select + time.After tunnel.go:178-193
High Ensure session disconnect explicitly cancels all spawned goroutines server_handler.go, tunnel.go

Reproduction

  1. Run chisel server with reverse tunnel: chisel server -R --reverse
  2. Connect a client with version mismatch (1.11.7 client → 1.11.8 server)
  3. Disconnect the client (network interruption or manual stop)
  4. Observe CPU usage climb as keepAliveLoop goroutines accumulate
  5. Reconnect repeatedly to accelerate the issue

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with share/tunnel/tunnel.go:178-193 and server/server_handler.go:84-94, then reproduce the disconnect and reconnect sequence described in the issue while observing CPU and goroutine behavior. Verify that closed-session handling stops the keep-alive work and avoids the nil request panic; done means repeated disconnects no longer accumulate runnable goroutines or cause the handler to panic.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Active
Clarity
Clearly specified
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.