Executing a command sometimes hangs forever
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 9.9k
- Forks
- 2.1k
- PR merge metrics
- No merged PRs in 30d
Description
I use paramiko to create SSH sessions and send commands to Linux machines, which works great 95% of the time. But sometimes, the Python thread which is used to execute the command hangs forever in wait() of threading.py:
(gdb) py-list
239 waiter.acquire()
240 self.__waiters.append(waiter)
241 saved_state = self._release_save()
242 try: # restore state no matter what (e.g., KeyboardInterrupt)
243 if timeout is None:
>244 waiter.acquire()
245 if __debug__:
246 self._note("%s.wait(): got it", self)
247 else:
248 # Balancing act: We can't afford a pure busy loop, so we
249 # have to sleep; but if we sleep the whole timeout time,
(gdb) py-bt
#5 Frame 0xb459a5c, for file /usr/lib/python2.7/threading.py, line 244, in wait (self=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e3a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e3a0>, _Condition__waiters=[<thread.lock at remote 0xaf7e100>], release=<built-in method release of thread.lock object at remote 0xaf7e3a0>) at remote 0xb73708c>, timeout=None, waiter=<thread.lock at remote 0xaf7e100>, saved_state=None)
waiter.acquire()
#9 Frame 0xb66118c, for file /usr/lib/python2.7/dist-packages/paramiko/buffered_pipe.py, line 137, in read (self=<BufferedPipe(_closed=False, _lock=<thread.lock at remote 0xaf7e3a0>, _buffer=<array.array at remote 0xb737ea0>, _event=None, _cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e3a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e3a0>, _Condition__waiters=[<thread.lock at remote 0xaf7e100>], release=<built-in method release of thread.lock object at remote 0xaf7e3a0>) at remote 0xb73708c>) at remote 0xb737a4c>, nbytes=8192, timeout=None, out='', then=<float at remote 0xb57efe4>)
self._cv.wait(timeout)
#13 Frame 0xb612b4c, for file /usr/lib/python2.7/dist-packages/paramiko/channel.py, line 584, in recv (self=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e4e0>) at remote 0xb7370ac>) at remote 0xb73732c>, transport=<Transport(_Thread__ident=-1258251408, host_key_type='ss...(truncated)
out = self.in_buffer.read(nbytes, self.timeout)
#16 Frame 0xa1875f4, for file /usr/lib/python2.7/dist-packages/paramiko/channel.py, line 1215, in _read (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], rele...(truncated)
return self.channel.recv(size)
#19 Frame 0xb8eb9dc, for file /usr/lib/python2.7/dist-packages/paramiko/file.py, line 214, in readline (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], relea...(truncated)
new_data = self._read(n)
#23 Frame 0xb4597f4, for file /usr/lib/python2.7/dist-packages/paramiko/file.py, line 106, in next (self=<ChannelFile(_wbuffer=<cStringIO.StringO at remote 0xb737860>, _closed=False, _bufsize=8192, _pos=105, _at_trailing_cr=False, _size=0, _rbuffer='', _flags=17, _realpos=105, channel=<Channel(origin_addr=None, in_window_size=65536, in_window_threshold=6553, lock=<thread.lock at remote 0xaf7e5a0>, _pipe=None, eof_received=0, in_max_packet_size=34816, out_buffer_cv=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e5a0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e5a0>, _Condition__waiters=[], release=<built-in method release of thread.lock object at remote 0xaf7e5a0>) at remote 0xb73790c>, event=<_Event(_Verbose__verbose=False, _Event__flag=True, _Event__cond=<_Condition(_Verbose__verbose=False, _Condition__lock=<thread.lock at remote 0xaf7e4e0>, acquire=<built-in method acquire of thread.lock object at remote 0xaf7e4e0>, _Condition__waiters=[], release=<...(truncated)
line = self.readline()
I have no idea whether this is an issue in paramiko or Python threading, so I also reported it there: http://bugs.python.org/issue24026
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading paramiko/buffered_pipe.py around BufferedPipe.read and paramiko/channel.py around Channel.recv, then compare the reported stack with Python issue 24026. Reproduce the intermittent SSH command hang and determine whether the wait is caused by Paramiko or Python threading; done means a confirmed cause and a scoped fix or documented external dependency.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- networking
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100