openwisp / openwisp/openwisp-controller

[bug] SSH exec_command throws away partial output and leaks channel on timeout

Open
#1,424 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug
Dominant language
Python
Stars
773
Forks
315
Avg merge
2d 16h
Merged PRs (30d)
14

Description

Describe the bug

When exec_command hits its SSH command timeout, any output the remote process sent before the timeout is silently discarded. The channel also remains open on the server side.

Three separate problems at openwisp_controller/connection/connectors/ssh.py:

  1. Partial output lost (lines 205-210). After status_event.wait() returns False, the method raises CommandTimeoutException without ever reading stdout or stderr:

    if not stdout.channel.status_event.wait(timeout=...):
        log_message = f"Command timed out after {timeout} seconds."
        logger.info(log_message)
        raise CommandTimeoutException(log_message)
    
  2. Channel leak (same block). No channel.close() is called, so the server-side session for the timed-out command is never released until the SSH transport is torn down.

  3. Traceback lost (lines 196-197). except socket.timeout: raise socket.timeout() creates a new exception instead of using bare raise, discarding the original traceback.
    => Not sure if this is intended or not ? if so I would add a comment next to it...

Steps To Reproduce

  1. Set up an SSH device connection with a remote host.
  2. Call connector_instance.exec_command("echo partial_output && sleep 60", timeout=0.5).
  3. Observe that CommandTimeoutException is raised with no .output attribute — "partial_output" is lost.
  4. Similar concerns for stderr...

Expected behavior

  • If the remote process managed to produce any output before the timeout, that output should be available to the caller (e.g., as the message or an attribute of the exception).
  • The SSH channel used for the timed-out command should be closed so the server can reclaim resources.
  • The original traceback should likely be preserved when socket.timeout propagates.

System Information:

  • OS: any
  • Python Version: any (3.x)
  • OpenWISP Controller version: master (v1.2.x)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in openwisp_controller/connection/connectors/ssh.py at exec_command and inspect the timeout handling around the referenced lines. Reproduce the issue with echo partial_output && sleep 60 and a 0.5-second timeout, then verify that partial stdout/stderr remains available, the timed-out channel is closed, and the original socket.timeout traceback is preserved.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.