Dashboard Server 500 Error

root@calamari01 ~]# /opt/calamari/venv/bin/cthulhu-manager
Traceback (most recent call last):
File "/opt/calamari/venv/bin/cthulhu-manager", line 9, in
load_entry_point('calamari-cthulhu==0.1', 'console_scripts', 'cthulhu-manager')()
File "/opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu/manager/manager.py", line 399, in main
m.start()
File "/opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu/manager/manager.py", line 322, in start
self._rpc_thread.bind()
File "/opt/calamari/venv/lib/python2.7/site-packages/calamari_cthulhu-0.1-py2.7.egg/cthulhu/manager/rpc.py", line 462, in bind
self._server.bind(config.get('cthulhu', 'rpc_url'))
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/socket.py", line 43, in bind
return self._events.bind(endpoint, resolve)
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/events.py", line 226, in bind
r.append(self._socket.bind(endpoint_))
File "socket.pyx", line 448, in zmq.backend.cython.socket.Socket.bind (zmq/backend/cython/socket.c:4127)
File "checkrc.pxd", line 21, in zmq.backend.cython.checkrc._check_rc (zmq/backend/cython/socket.c:6083)
zmq.error.ZMQError: Address already in use

==> calamari/calamari.log <==
2015-11-04 12:57:38,596 - ERROR - django.request Internal Server Error: /dashboard/
Traceback (most recent call last):
File "/opt/calamari/venv/lib/python2.7/site-packages/django/core/handlers/base.py", line 115, in get_response
response = callback(request, _callback_args, *_callback_kwargs)
File "/opt/calamari/venv/lib/python2.7/site-packages/django/contrib/auth/decorators.py", line 25, in _wrapped_view
return view_func(request, _args, *_kwargs)
File "/opt/calamari/venv/lib/python2.7/site-packages/calamari_web-0.1-py2.7.egg/calamari_web/views.py", line 38, in dashboard
clusters = client.list_clusters()
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 260, in
return lambda _args, *_kargs: self(method, _args, *_kargs)
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 245, in __call__
return self._process_response(request_event, bufchan, timeout)
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/core.py", line 217, in _process_response
reply_event = bufchan.recv(timeout)
File "/opt/calamari/venv/lib/python2.7/site-packages/zerorpc/channel.py", line 267, in recv
event = self._input_queue.get(timeout=timeout)
File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/queue.py", line 200, in get
result = waiter.get()
File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/hub.py", line 575, in get
return self.hub.switch()
File "/opt/calamari/venv/lib/python2.7/site-packages/gevent/hub.py", line 338, in switch
return greenlet.switch(self)
LostRemote: Lost remote after 10s heartbeat

[root@calamari01 ~]# yum list salt*
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
- base: mirror.cogentco.com
- epel: mirror.cogentco.com
- extras: mirror.us.leaseweb.net
- updates: mirror.umd.edu
Installed Packages
salt.noarch 2015.5.5-1.el7 @epel
salt-master.noarch 2015.5.5-1.el7 @epel
salt-minion.noarch 2015.5.5-1.el7 @epel

[root@calamari01 ~]# rpm -qa | grep calamari
calamari-server-1.3.1.1-101_g945d16a.el7.centos.x86_64

[root@calamari01 ~]# salt-key -L
Accepted Keys:
Denied Keys:
Unaccepted Keys:
ceph1
Rejected Keys:

built packages using these instructions http://karan-mj.blogspot.fi/2014/09/ceph-calamari-survival-guide.html

I've tried installing calamari on centos 6 & 7 using either packages built from git or from https://github.com/ksingh7/ceph-calamari-packages. No matter what I do as soon as a ceph node sends it's salt key I get 500 errors. I've tried salt 2014 on centos 6 as well.

Start with cthulhu/manager/manager.py and cthulhu/manager/rpc.py around the RPC bind failure, then inspect calamari_web/views.py at the dashboard view and the supplied calamari.log traceback. Reproduce the Salt key acceptance flow and determine what must change so the manager remains reachable and /dashboard/ no longer returns HTTP 500.

Written by the indexing model from the issue text.