paramiko / paramiko/paramiko

Executing a command sometimes hangs forever

Open
#515 39 comments 6 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug Hangs Needs investigation
Dominant language
Python
Stars
9.9k
Forks
2.1k
PR merge metrics
No merged PRs in 30d

Description

I use paramiko to create SSH sessions and send commands to Linux machines, which works great 95% of the time. But sometimes, the Python thread which is used to execute the command hangs forever in wait() of threading.py:

(gdb) py-list
 239            waiter.acquire()
 240            self.__waiters.append(waiter)
 241            saved_state = self._release_save()
 242            try:    # restore state no matter what (e.g., KeyboardInterrupt) 
 243                if timeout is None:
>244                    waiter.acquire()
 245                    if __debug__:
 246                        self._note("%s.wait(): got it", self)
 247                else:
 248                    # Balancing act:  We can't afford a pure busy loop, so we
 249                    # have to sleep; but if we sleep the whole timeout time,


(gdb) py-bt
#5 Frame 0xb459a5c, for file /usr/lib/python2.7/threading.py, line 244, in wait (self=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e3a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e3a0>, _Condition__waiters=[<thread.lock at remote 0xaf7e100>], release=<built-in method release of thread.lock object at remote 0xaf7e3a0>) at remote 0xb73708c>, timeout=None, waiter=<thread.lock at remote 0xaf7e100>, saved_state=None)
    waiter.acquire()
#9 Frame 0xb66118c, for file /usr/lib/python2.7/dist-packages/paramiko/buffered_pipe.py, line 137, in read (self=<BufferedPipe(_closed=False, _lock=<thread.lock at remote 0xaf7e3a0>, _buffer=<array.array at remote 0xb737ea0>, _event=None, _cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e3a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e3a0>, _Condition__waiters=[<thread.lock at remote 0xaf7e100>], release=<built-in method release of thread.lock object at remote 0xaf7e3a0>) at remote 0xb73708c>) at remote 0xb737a4c>, nbytes=8192, timeout=None, out='', then=<float at remote 0xb57efe4>)
    self._cv.wait(timeout)
#13 Frame 0xb612b4c, for file /usr/lib/python2.7/dist-packages/paramiko/channel.py, line 584, in recv (self=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e4e0>) at remote 0xb7370ac>) at remote 0xb73732c>, transport=<Transport(_Thread__ident=-1258251408, host_key_type='ss...(truncated)
    out = self.in_buffer.read(nbytes, self.timeout)
#16 Frame 0xa1875f4, for file /usr/lib/python2.7/dist-packages/paramiko/channel.py, line 1215, in _read (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], rele...(truncated)
    return self.channel.recv(size)
#19 Frame 0xb8eb9dc, for file /usr/lib/python2.7/dist-packages/paramiko/file.py, line 214, in readline (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], relea...(truncated)
    new_data = self._read(n)
#23 Frame 0xb4597f4, for file /usr/lib/python2.7/dist-packages/paramiko/file.py, line 106, in next (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], release=<...(truncated)
    line = self.readline()

I have no idea whether this is an issue in paramiko or Python threading, so I also reported it there: http://bugs.python.org/issue24026

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading paramiko/buffered_pipe.py around BufferedPipe.read and paramiko/channel.py around Channel.recv, then compare the reported stack with Python issue 24026. Reproduce the intermittent SSH command hang and determine whether the wait is caused by Paramiko or Python threading; done means a confirmed cause and a scoped fix or documented external dependency.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.