StackStorm / StackStorm/st2

Eventlet causing issues in thread pools for sensors

Open
#3,818 2 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

stale status:to be verified
Dominant language
Python
Stars
6.5k
Forks
787
PR merge metrics
No merged PRs in 30d

Description

See https://github.com/eventlet/eventlet/issues/147

Observed with msexchange pack, which uses requests when watching the sensor container directly.

[root@irtyrt1 msexchange]# /opt/stackstorm/st2/bin/st2sensorcontainer --config-file=/etc/st2/st2.conf --sensor-ref=msexchange.ItemSensor
2017-10-30 03:52:44,287 DEBUG [-] Using config files: /etc/st2/st2.conf
2017-10-30 03:52:44,287 DEBUG [-] Using logging config: /etc/st2/logging.sensorcontainer.conf
2017-10-30 03:52:44,289 INFO [-] Connecting to database "st2" @ "127.0.0.1:27017" as user "stackstorm".
2017-10-30 03:52:44,454 INFO [-] Setting up container to run 1 sensors.
2017-10-30 03:52:44,454 INFO [-]  Sensors list - [u'msexchange.ItemSensor'].
2017-10-30 03:52:44,455 INFO [-] (PID:25913) SensorContainer started.
2017-10-30 03:52:44,455 INFO [-] Running sensor msexchange.ItemSensor
2017-10-30 03:52:44,466 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:44,466 AUDIT [-] Access granted to "sensors_container" with the token set to expire at "2017-10-31T07:52:44.464372Z". (username='sensors_container',token_expiration='2017-10-31T07:52:44.464372Z')
2017-10-30 03:52:44,467 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:44,504 INFO [-] Sensor msexchange.ItemSensor started
2017-10-30 03:52:45,401 INFO [-] Found config for sensor "ItemSensor"
2017-10-30 03:52:45,411 INFO [-] Watcher started
2017-10-30 03:52:45,411 INFO [-] Running sensor initialization code
2017-10-30 03:52:45,431 INFO [-] Connected to amqp://guest:**@127.0.0.1:5672//
2017-10-30 03:52:47,016 INFO [-] API version "Exchange2007" worked but server reports version "Exchange2016". Using "Exchange2016"
2017-10-30 03:52:47,019 ERROR [-] Exception in thread Thread-17:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 301, in _handle_workers
    while thread._state == RUN or (pool._cache and thread._state != TERMINATE):
AttributeError: '_MainThread' object has no attribute '_state'


2017-10-30 03:52:47,101 INFO [-] Running sensor in passive mode
2017-10-30 03:52:47,101 AUDIT [-] Creating new Client object.
2017-10-30 03:52:47,106 AUDIT [-] Access granted to "sensor_service" with the token set to expire at "2017-10-31T07:52:47.104604Z". (username='sensor_service',token_expiration='2017-10-31T07:52:47.104604Z')
2017-10-30 03:52:47,106 WARNING [-] "auth.api_url" configuration option is not configured
2017-10-30 03:52:47,106 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:47,130 ERROR [-] /opt/stackstorm/virtualenvs/msexchange/lib/python2.7/site-packages/exchangelib/folders.py:127: UserWarning: The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" or "some_folder / 'Sub Folder'" instead, to find folders by name.
  warnings.warn('The get_folder_by_name() method is deprecated. Use "[f for f in self.walk() if f.name == name]" '

2017-10-30 03:52:47,585 INFO [-] Found 0 items
2017-10-30 03:52:52,668 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:52,819 INFO [-] Found 0 items
2017-10-30 03:52:57,901 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:52:58,042 INFO [-] Found 0 items
2017-10-30 03:53:03,125 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:03,272 INFO [-] Found 0 items
2017-10-30 03:53:08,352 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:08,498 INFO [-] Found 0 items
2017-10-30 03:53:13,578 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:13,719 INFO [-] Found 0 items
2017-10-30 03:53:18,800 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:18,942 INFO [-] Found 0 items
2017-10-30 03:53:24,025 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:24,166 INFO [-] Found 0 items
2017-10-30 03:53:29,246 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:29,399 INFO [-] Found 0 items
2017-10-30 03:53:34,480 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:34,622 INFO [-] Found 0 items
2017-10-30 03:53:39,703 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:39,844 INFO [-] Found 0 items
2017-10-30 03:53:44,933 AUDIT [-] Retrieving value from the datastore (name=msexchange.ItemSensor:exchange.item_sensor_date_str)
2017-10-30 03:53:45,076 INFO [-] Found 1 items
2017-10-30 03:53:45,157 ERROR [-] Exception in thread Thread-18:
Traceback (most recent call last):
  File "/usr/lib64/python2.7/threading.py", line 812, in __bootstrap_inner
    self.run()
  File "/usr/lib64/python2.7/threading.py", line 765, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/usr/lib64/python2.7/multiprocessing/pool.py", line 315, in _handle_tasks
    if thread._state:
AttributeError: '_MainThread' object has no attribute '_state'

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at the st2sensorcontainer entry point and inspect how sensors using requests are run with Eventlet and multiprocessing thread pools. Reproduce the failure with the shown sensor-container command, then compare the behavior with Eventlet issue 147. Done means the thread-pool worker threads no longer raise the reported _MainThread _state errors while the sensor continues polling.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.