StackStorm / StackStorm/st2

st2api service fails to restart after adding configuration for backend redis service

Open
#2,640 2 comments 0 reactions 1 assignee View on GitHub

@lakshmi-kannan is already working on this.

Since Nov 22, 2016.

enhancement
Dominant language
Python
Stars
6.5k
Forks
787
PR merge metrics
No merged PRs in 30d

Description

Using the Stackstorm 1.3 package-based install, I went in and configured redis in /etc/st2/st2.conf per the policies docs. This seemed to be fine initially but eventually something must have happened to cause the st2api service to restart and then (according to systemctl -l status st2api.service output) it looks like it got into some kind of restart loop.

I finally caught the offending output (this was pretty difficult to do and doesn't appear to be making its way into st2api.log), looks like the /opt/stackstorm/st2 Python virtualenv was missing the redis module:

st2api.service - StackStorm service st2api
Loaded: loaded (/usr/lib/systemd/system/st2api.service; enabled; vendor preset: disabled)
Active: active (running) since Thu 2016-04-21 10:42:01 EDT; 28s ago
Main PID: 26270 (gunicorn_pecan)
CGroup: /system.slice/st2api.service
\u251c\u250026270 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py -k eventlet -b 127.0.0.1:9101 --threads 10 --workers 1 --graceful-timeout 10
\u2514\u250026522 /opt/stackstorm/st2/bin/python /opt/stackstorm/st2/bin/gunicorn_pecan /opt/stackstorm/st2/lib/python2.7/site-packages/st2api/gunicorn_config.py -k eventlet -b 127.0.0.1:9101 --threads 10 --workers 1 --graceful-timeout 10

Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/stevedore/named.py", line 123, in _load_one_plugin
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: verify_requirements,
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/stevedore/extension.py", line 183, in _load_one_plugin
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: plugin = ep.resolve()
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/pkg_resources/init.py", line 2208, in resolve
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: module = import(self.module_name, fromlist=['name'], level=0)
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: File "/opt/stackstorm/st2/lib/python2.7/site-packages/tooz/drivers/redis.py", line 27, in
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: import redis
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: ImportError: No module named redis
Apr 21 10:42:30 pxops-st2 gunicorn_pecan[26270]: [2016-04-21 10:42:30 +0000] [26522] [INFO] Worker exiting (pid: 26522)

My fix was to install the redis module into the virtualenv, ie. sudo su; source /opt/stackstorm/st2/bin/activate; pip install redis

After this systemctl restart st2api appeared to work and everything was happy again.

Not sure if this is a missing dependency in packaging or a setup script or just needs to be noted in the docs. Let me know if you need any more info about my setup.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.