Eventlet causing issues in thread pools for sensors
Open
Nobody has claimed this yet.
stale
status:to be verified
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 787
- PR merge metrics
- No merged PRs in 30d
Description
See https://github.com/eventlet/eventlet/issues/147
Observed with msexchange pack, which uses requests when watching the sensor container directly.
[root@irtyrt1 msexchange]# /opt/stackstorm/st2/bin/st2sensorcontainer --config-file=/etc/st2/st2.conf --sensor-ref=msexchange.ItemSensor
2017-10-30 03:52:44,287 DEBUG [-] Using config files: /etc/st2/st2.conf
2017-10-30 03:52:44,287 DEBUG [-] Using logging config: /etc/st2/logging.sensorcontainer.conf
2017-10-30 03:52:44,289 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2017-10-30 03:52:44,454 INFO [-] Setting up container to run 1 sensors.
2017-10-30 03:52:44,454 INFO [-] Sensors list - [u'msexchange.ItemSensor'].
2017-10-30 03:52:44,455 INFO [-] (PID:25913) SensorContainer started.
2017-10-30 03:52:44,455 INFO [-] Running sensor msexchange.ItemSensor
2017-10-30 03:52:44,466 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:44,466 AUDIT [-] Access granted to "sensors_container" with the token set to expire at "2017-10-31T07:52:44.464372Z". (username='sensors_container',token_expiration='2017-10-31T07:52:44.464372Z')
2017-10-30 03:52:44,467 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:44,504 INFO [-] Sensor msexchange.ItemSensor started
2017-10-30 03:52:45,401 INFO [-] Found config for sensor "ItemSensor"
2017-10-30 03:52:45,411 INFO [-] Watcher started
2017-10-30 03:52:45,411 INFO [-] Running sensor initialization code
2017-10-30 03:52:45,431 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:47,016 INFO [-] API version "Exchange2007" worked but server reports version "Exchange2016". Using "Exchange2016"
2017-10-30 03:52:47,019 ERROR [-] Exception in thread Thread-17:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 301, in _handle_workers
while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'
2017-10-30 03:52:47,101 INFO [-] Running sensor in passive mode
2017-10-30 03:52:47,101 AUDIT [-] Creating new Client object.
2017-10-30 03:52:47,106 AUDIT [-] Access granted to "sensor_service" with the token set to expire at "2017-10-31T07:52:47.104604Z". (username='sensor_service',token_expiration='2017-10-31T07:52:47.104604Z')
2017-10-30 03:52:47,106 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:47,106 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:47,130 ERROR [-] /opt/stackstorm/virtualenvs/msexchange/lib/python2.7/site-packages/exchangelib/folders.py:127: UserWarning: The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" or "some_folder / 'Sub Folder'" instead, to find folders by name.
warnings.warn('The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" '
2017-10-30 03:52:47,585 INFO [-] Found 0 items
2017-10-30 03:52:52,668 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:52,819 INFO [-] Found 0 items
2017-10-30 03:52:57,901 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:58,042 INFO [-] Found 0 items
2017-10-30 03:53:03,125 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:03,272 INFO [-] Found 0 items
2017-10-30 03:53:08,352 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:08,498 INFO [-] Found 0 items
2017-10-30 03:53:13,578 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:13,719 INFO [-] Found 0 items
2017-10-30 03:53:18,800 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:18,942 INFO [-] Found 0 items
2017-10-30 03:53:24,025 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:24,166 INFO [-] Found 0 items
2017-10-30 03:53:29,246 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:29,399 INFO [-] Found 0 items
2017-10-30 03:53:34,480 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:34,622 INFO [-] Found 0 items
2017-10-30 03:53:39,703 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:39,844 INFO [-] Found 0 items
2017-10-30 03:53:44,933 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:45,076 INFO [-] Found 1 items
2017-10-30 03:53:45,157 ERROR [-] Exception in thread Thread-18:
Traceback (most recent call last):
File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
self.run()
File "/usr/lib64/python2.7/threading.py", line 765, in run
self.__target(*self.__args, **self.__kwargs)
File "/usr/lib64/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
if thread._state:
AttributeError: '_MainThread' object has no attribute '_state'
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at the st2sensorcontainer entry point and inspect how sensors using requests are run with Eventlet and multiprocessing thread pools. Reproduce the failure with the shown sensor-container command, then compare the behavior with Eventlet issue 147. Done means the thread-pool worker threads no longer raise the reported _MainThread _state errors while the sensor continues polling.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100