ST2 3.5 rpm not creating admin roles in mongo; other services not connecting to mongo
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 6.5k
- Forks
- 787
- PR merge metrics
- No merged PRs in 30d
Description
SUMMARY
Running ST 3.5 rpm’s, Centos8, Mongo 4, etc..
At startup, st2-apply-rbac-definitions does not insert the admin role to mongo. Subsequent user assignments (stanley) can’t be loaded as the role doesn’t exist. I can switch back to the 3.4.1 rpm’s and it is not an issue.
In an oddly semi-related issue, the above st2-apply-rbac-definitions job can connect to Mongo and load all the other user assignments without issue.
However, none of the other services that start (actionrunner, etc..) can connect to Mongo (all the same settings). All reporting pymongo.errors.ServerSelectionTimeoutError: module 'select' has no attribute 'poll'
STACKSTORM VERSION
Paste the output of st2 --version:
3.5.0-1 rpm's
OS, environment, install method
Post what OS you are running this on, along with any other relevant information/
Kubernetes HA charts, Docker, CentOS8
Steps to reproduce the problem
- st2-apply-rbac-definitions
- --verbose
- --config-file=/etc/st2/st2.conf
- --config-file=/etc/st2/st2.docker.conf
- --config-file=/etc/st2/st2.user.conf
start up actionrunner or other service
Expected Results
results with mongo db.role_d_b.find() shows no admin, system_admin, observer roles
actionrunner service to start
Actual Results
no admin roles; cannot assign stanley to admin role
pymongo.errors.ServerSelectionTimeoutError: module 'select' has no attribute 'poll'
Full error:
st2actionrunner 2021-07-12 20:11:54,319 INFO [-] Using Python: 3.6.8 (/opt/stackstorm/st2/bin/python)
st2actionrunner 2021-07-12 20:11:54,320 INFO [-] Using fs encoding: utf-8, default encoding: utf-8, locale: en_US.UTF-8, LANG env variable: en_US.UTF-8, PYTHONIOENCODING env variable: notset
st2actionrunner 2021-07-12 20:11:54,320 INFO [-] Using config files: /etc/st2/st2.conf,/etc/st2/st2.docker.conf,/etc/st2/st2.user.conf
st2actionrunner 2021-07-12 20:11:54,321 INFO [-] Using logging config: /etc/st2/logging.docker.conf
st2actionrunner 2021-07-12 20:11:54,321 INFO [-] Using coordination driver: redis
st2actionrunner 2021-07-12 20:11:54,321 INFO [-] Using metrics driver: statsd
st2actionrunner 2021-07-12 20:11:54,326 INFO [-] Connecting to database "st2" @ "mongodb-0.mongodb-headless:27017,mongodb-1.mongodb-headless:27017,mongodb-2.mongodb-headless:27017 (replica set)" as user "s
st2actionrunner 2021-07-12 20:11:57,376 ERROR [-] Failed to connect to database "st2" @ "mongodb-0.mongodb-headless:27017,mongodb-1.mongodb-headless:27017,mongodb-2.mongodb-headless:27017 (replica set)" as
st2actionrunner 2021-07-12 20:11:57,376 ERROR [-] (PID=1) Worker quit due to exception.
st2actionrunner Traceback (most recent call last):
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 98, in main
st2actionrunner _setup()
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2actions/cmd/actionrunner.py", line 58, in _setup
st2actionrunner capabilities=capabilities,
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/service_setup.py", line 244, in setup
st2actionrunner db_setup()
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/database_setup.py", line 55, in db_setup
st2actionrunner connection = db_init.db_setup_with_retry(**db_cfg)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 93, in db_setup_with_retry
st2actionrunner ssl_match_hostname=ssl_match_hostname,
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/persistence/db_init.py", line 58, in db_func_with_retry
st2actionrunner return retrying_obj.call(db_func, *args, **kwargs)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 206, in call
st2actionrunner return attempt.get(self._wrap_exception)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 247, in get
st2actionrunner six.reraise(self.value[0], self.value[1], self.value[2])
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/six.py", line 696, in reraise
st2actionrunner raise value
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/retrying.py", line 200, in call
st2actionrunner attempt = Attempt(fn(*args, **kwargs), attempt_number, False)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 250, in db_setup
st2actionrunner ssl_match_hostname=ssl_match_hostname,
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 212, in _db_connect
st2actionrunner raise e
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/st2common/models/db/__init__.py", line 203, in _db_connect
st2actionrunner connection.admin.command("ismaster")
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/database.py", line 737, in command
st2actionrunner read_preference, session) as (sock_info, slave_ok):
st2actionrunner File "/usr/lib64/python3.6/contextlib.py", line 81, in __enter__
st2actionrunner return next(self.gen)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1325, in _socket_for_reads
st2actionrunner server = self._select_server(read_preference, session)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/mongo_client.py", line 1278, in _select_server
st2actionrunner server = topology.select_server(server_selector)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 243, in select_server
st2actionrunner address))
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 200, in select_servers
st2actionrunner selector, server_timeout, address)
st2actionrunner File "/opt/stackstorm/st2/lib/python3.6/site-packages/pymongo/topology.py", line 217, in _select_servers_loop
st2actionrunner (self._error_message(selector), timeout, self.description))
st2actionrunner pymongo.errors.ServerSelectionTimeoutError: module 'select' has no attribute 'poll',module 'select' has no attribute 'poll',module 'select' has no attribute 'poll', Timeout: 3.0s, Topology Description: <T
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the st2-apply-rbac-definitions entry point and the service paths named in st2actions/cmd/actionrunner.py, st2common/database_setup.py, st2common/persistence/db_init.py, and st2common/models/db/init.py. Reproduce the issue on the listed StackStorm 3.5 environment and compare with 3.4.1. Done means the default roles are created in MongoDB and actionrunner and other services connect successfully.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- centos, docker, kubernetes, mongodb, python
- Domain
- backend, databases, devops
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100