twisted / twisted/twisted

t.i.serialport.Serialport().flushInput() causes exception on Windows

Open
#9,720 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

bug core new priority-normal
Dominant language
Python
Stars
6k
Forks
1.2k
Avg merge
2d 10h
Merged PRs (30d)
10

Description

altendky's avatar @altendky reported
Trac ID trac#9720
Type defect
Created 2019-10-24 00:06:59Z

Flushing serial input on Windows causes the twisted internal read operations to be canceled. Given that the flush/reset is being requested through twisted it seems like we should be able to handle this temporarily expected exception. We may also want to clear some internal buffers, I don't recall how that works at the moment. I'll see if I can come up with a sensible PR.

pywintypes.error: (995, 'GetOverlappedResult', 'The I/O operation has been aborted because of either a thread exit or an application request.')

Flushing in pyserial has long been doing the same by passing the abort flag.

2002: https://github.com/pyserial/pyserial/commit/89b4af15d858f4289c5e1c8040b171cbe95c4e6e#diff-8afcad5e5ea03f7f8d208bc147db67e2R260

win32file.PurgeComm(self.hComPort, win32file.PURGE_RXCLEAR | win32file.PURGE_RXABORT)

2019: https://github.com/pyserial/pyserial/commit/98b8c1a7df33591f8b1753388ba1a06d9f93db80#diff-8dcb5a1563d53133df161c6a5f664916

win32.PurgeComm(self._port_handle, win32.PURGE_RXCLEAR | win32.PURGE_RXABORT)

Here's the log of creating the issue with the contrived serial.win32.PurgeComm() call.

PS C:\epc\test-software> py -3.7-64 -m venv venv

PS C:\epc\test-software> venv/scripts/pip install twisted pywin32 pyserial
Collecting twisted
  Using cached https://files.pythonhosted.org/packages/ee/d9/5b79fef4a7d7dc4d526151904eae5dd207f80433ae646a258b32abbe77d4/Twisted-19.7.0-cp37-cp37m-win_amd64.whl
Collecting pywin32
  Using cached https://files.pythonhosted.org/packages/b5/26/801bc6d99612977accac84b2d8ad06efa4a7d831c5e2813ca9baf700b646/pywin32-225-cp37-cp37m-win_amd64.whl
Collecting pyserial
  Using cached https://files.pythonhosted.org/packages/0d/e4/2a744dd9e3be04a0c0907414e2a01a7c88bb3915cbe3c8cc06e209f59c30/pyserial-3.4-py2.py3-none-any.whl
Collecting incremental>=16.10.1 (from twisted)
  Using cached https://files.pythonhosted.org/packages/f5/1d/c98a587dc06e107115cf4a58b49de20b19222c83d75335a192052af4c4b7/incremental-17.5.0-py2.py3-none-any.whl
Collecting PyHamcrest>=1.9.0 (from twisted)
  Using cached https://files.pythonhosted.org/packages/9a/d5/d37fd731b7d0e91afcc84577edeccf4638b4f9b82f5ffe2f8b62e2ddc609/PyHamcrest-1.9.0-py2.py3-none-any.whl
Collecting constantly>=15.1 (from twisted)
  Using cached https://files.pythonhosted.org/packages/b9/65/48c1909d0c0aeae6c10213340ce682db01b48ea900a7d9fce7a7910ff318/constantly-15.1.0-py2.py3-none-any.whl
Collecting hyperlink>=17.1.1 (from twisted)
  Using cached https://files.pythonhosted.org/packages/7f/91/e916ca10a2de1cb7101a9b24da546fb90ee14629e23160086cf3361c4fb8/hyperlink-19.0.0-py2.py3-none-any.whl
Collecting attrs>=17.4.0 (from twisted)
  Using cached https://files.pythonhosted.org/packages/a2/db/4313ab3be961f7a763066401fb77f7748373b6094076ae2bda2806988af6/attrs-19.3.0-py2.py3-none-any.whl
Collecting zope.interface>=4.4.2 (from twisted)
  Using cached https://files.pythonhosted.org/packages/a8/d2/e2fb1052cdf1c1d05a23c5f7a192a8dc104d5afda0539f86b9839264e1cc/zope.interface-4.6.0-cp37-cp37m-win_amd64.whl
Collecting Automat>=0.3.0 (from twisted)
  Using cached https://files.pythonhosted.org/packages/e5/11/756922e977bb296a79ccf38e8d45cafee446733157d59bcd751d3aee57f5/Automat-0.8.0-py2.py3-none-any.whl
Collecting six (from PyHamcrest>=1.9.0->twisted)
  Using cached https://files.pythonhosted.org/packages/73/fb/00a976f728d0d1fecfe898238ce23f502a721c0ac0ecfedb80e0d88c64e9/six-1.12.0-py2.py3-none-any.whl
Requirement already satisfied: setuptools in c:\epc\test-software\venv\lib\site-packages (from PyHamcrest>=1.9.0->twisted) (40.8.0)
Collecting idna>=2.5 (from hyperlink>=17.1.1->twisted)
  Using cached https://files.pythonhosted.org/packages/14/2c/cd551d81dbe15200be1cf41cd03869a46fe7226e7450af7a6545bfc474c9/idna-2.8-py2.py3-none-any.whl
Installing collected packages: incremental, six, PyHamcrest, constantly, idna, hyperlink, attrs, zope.interface, Automat, twisted, pywin32, pyserial
Successfully installed Automat-0.8.0 PyHamcrest-1.9.0 attrs-19.3.0 constantly-15.1.0 hyperlink-19.0.0 idna-2.8 incremental-17.5.0 pyserial-3.4 pywin32-225 six-1.12.0 twisted-19.7.0 zope.interface-4.6.0
You are using pip version 19.0.3, however version 19.3.1 is available.
You should consider upgrading via the 'python -m pip install --upgrade pip' command.

PS C:\epc\test-software> venv/scripts/python --version --version
Python 3.7.4 (tags/v3.7.4:e09359112e, Jul  8 2019, 20:34:20) [MSC v.1916 64 bit (AMD64)]

PS C:\epc\test-software> venv/scripts/pip freeze
attrs==19.3.0
Automat==0.8.0
constantly==15.1.0
hyperlink==19.0.0
idna==2.8
incremental==17.5.0
PyHamcrest==1.9.0
pyserial==3.4
pywin32==225
six==1.12.0
Twisted==19.7.0
zope.interface==4.6.0

First the non-failing output.

PS C:\epc\test-software> venv/scripts/python example.py
  ---- CHECKPOINT: a
  ---- CHECKPOINT: b
  ---- CHECKPOINT: c

Here is the basic failure.

PS C:\epc\test-software> venv/scripts/python example.py --fail
  ---- CHECKPOINT: a
  ---- CHECKPOINT: b
Unhandled Error
Traceback (most recent call last):
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1283, in run
    self.mainLoop()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1292, in mainLoop
    self.runUntilCurrent()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 886, in runUntilCurrent
    f(*a, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 117, in _callFromThread
    result = defer.maybeDeferred(f, *a, **kw)
--- <exception caught here> ---
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 300, in _runAction
    closed = getattr(fd, action)()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in _execute
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\python\failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in <lambda>
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\_win32serialport.py", line 80, in serialReadEvent
    n = win32file.GetOverlappedResult(self._serial._port_handle, self._overlappedRead, 0)
pywintypes.error: (995, 'GetOverlappedResult', 'The I/O operation has been aborted because of either a thread exit or an application request.')

  ---- CHECKPOINT: c

And another mode of failure that additionally includes a couple builtins.KeyError exceptions.

PS C:\epc\test-software> venv/scripts/python example.py --fail
  ---- CHECKPOINT: a
  ---- CHECKPOINT: b
Unhandled Error
Traceback (most recent call last):
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1283, in run
    self.mainLoop()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1292, in mainLoop
    self.runUntilCurrent()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 886, in runUntilCurrent
    f(*a, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 117, in _callFromThread
    result = defer.maybeDeferred(f, *a, **kw)
--- <exception caught here> ---
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 300, in _runAction
    closed = getattr(fd, action)()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in _execute
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\python\failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in <lambda>
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\_win32serialport.py", line 80, in serialReadEvent
    n = win32file.GetOverlappedResult(self._serial._port_handle, self._overlappedRead, 0)
pywintypes.error: (995, 'GetOverlappedResult', 'The I/O operation has been aborted because of either a thread exit or an application request.')

Unhandled Error
Traceback (most recent call last):
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1283, in run
    self.mainLoop()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1292, in mainLoop
    self.runUntilCurrent()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 886, in runUntilCurrent
    f(*a, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 117, in _callFromThread
    result = defer.maybeDeferred(f, *a, **kw)
--- <exception caught here> ---
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 300, in _runAction
    closed = getattr(fd, action)()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in _execute
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\threads.py", line 122, in blockingCallFromThread
    result.raiseException()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\python\failure.py", line 488, in raiseException
    raise self.value.with_traceback(self.tb)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\defer.py", line 151, in maybeDeferred
    result = f(*args, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 349, in <lambda>
    self._reactor, lambda: getattr(self._fd, self._action)())
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\_win32serialport.py", line 80, in serialReadEvent
    n = win32file.GetOverlappedResult(self._serial._port_handle, self._overlappedRead, 0)
pywintypes.error: (995, 'GetOverlappedResult', 'The I/O operation has been aborted because of either a thread exit or an application request.')

Unhandled Error
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1283, in run
    self.mainLoop()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1292, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 886, in runUntilCurrent
    f(*a, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 154, in removeEvent
    del self._events[event]
builtins.KeyError: <PyHANDLE:780>

Unhandled Error
Traceback (most recent call last):
  File "C:\Program Files\Python37\lib\threading.py", line 926, in _bootstrap_inner
    self.run()
  File "C:\Program Files\Python37\lib\threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1283, in run
    self.mainLoop()
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 1292, in mainLoop
    self.runUntilCurrent()
--- <exception caught here> ---
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\base.py", line 886, in runUntilCurrent
    f(*a, **kw)
  File "C:\epc\test-software\venv\lib\site-packages\twisted\internet\win32eventreactor.py", line 154, in removeEvent
    del self._events[event]
builtins.KeyError: <PyHANDLE:708>

  ---- CHECKPOINT: c

PS C:\epc\test-software> type example.py

import argparse
import sys

import twisted.internet.defer
import twisted.internet.serialport
import twisted.internet.task
import serial.win32


@twisted.internet.defer.inlineCallbacks
def go(reactor, port, baud_rate, parity, fail):
    protocol = twisted.internet.protocol.Protocol()

    port = twisted.internet.serialport.SerialPort(
        protocol=protocol,
        deviceNameOrPortNumber=port,
        reactor=reactor,
        baudrate=baud_rate,
        bytesize=8,
        parity=parity,
        stopbits=1,
        xonxoff=False,
        rtscts=False,
    )

    print('  ---- CHECKPOINT: a', file=sys.stderr)
    # port.flushInput() / port._serial.reset_input_buffer()
    flags = serial.win32.PURGE_RXCLEAR

    if fail:
        flags |= serial.win32.PURGE_RXABORT

    serial.win32.PurgeComm(
        port._serial._port_handle,
        flags,
    )
    print('  ---- CHECKPOINT: b', file=sys.stderr)
    yield twisted.internet.task.deferLater(
        twisted.internet.reactor,
        1,
        lambda: None,
    )
    print('  ---- CHECKPOINT: c', file=sys.stderr)


def main():
    parser = argparse.ArgumentParser()
    parser.add_argument('--port', default='com4')
    parser.add_argument('--baud-rate', type=int, default=19200)
    parser.add_argument('--parity', default='N')
    parser.add_argument('--fail', action='store_true')

    args = parser.parse_args()

    twisted.internet.task.react(
        go,
        (args.port, args.baud_rate, args.parity, args.fail),
    )


main()
Searchable metadata 
trac-id__9720 9720
type__defect defect
reporter__altendky altendky
priority__normal normal
milestone__None None
branch__ 
branch_author__ 
status__new new
resolution__None None
component__core core
keywords__None None
time__1571875619218458 1571875619218458
changetime__1571875647116033 1571875647116033
version__None None
owner__altendky altendky

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with twisted/internet/_win32serialport.py at serialReadEvent and follow the related win32eventreactor.py paths shown in the traceback. Reproduce the Windows flushInput case using the reported failure mode, then verify that the expected canceled read does not produce unhandled errors or leave inconsistent internal event state.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
networking, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.