galaxyproject / galaxyproject/pulsar

Restarting RESTful Pulsar is not safe

Closed
#114 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
46
Forks
62
Avg merge
3d 9h
Merged PRs (30d)
15

Description

``` pytb
galaxy.jobs.runners.pulsar ERROR 2016-05-20 12:54:43,326 failure finishing job 544006
Traceback (most recent call last):
File "/cvmfs/test.galaxyproject.org/galaxy/lib/galaxy/jobs/runners/pulsar.py", line 452, in finish_job
run_results = client.full_status()
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/client.py", line 133, in full_status
return self.raw_check_complete()
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/decorators.py", line 28, in replacement
return func(*args, **kwargs)
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/decorators.py", line 13, in replacement
response = func(*args, **kwargs)
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/client.py", line 147, in raw_check_complete
check_complete_response = self._raw_execute("status", {"job_id": self.job_id})
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/client.py", line 218, in _raw_execute
return self.job_manager_interface.execute(command, args, data, input_path, output_path)
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/interface.py", line 98, in execute
response = self.transport.execute(url, method=method, data=data, input_path=input_path, output_path=output_path)
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/transport/standard.py", line 34, in execute
response = self._url_open(request, data)
File "/cvmfs/test.galaxyproject.org/venv/lib/python2.7/site-packages/pulsar/client/transport/standard.py", line 20, in _url_open
return urlopen(request, data)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 154, in urlopen
return opener.open(url, data, timeout)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 431, in open
response = self._open(req, data)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 449, in _open
'_open', req)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 409, in _call_chain
result = func(*args)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 1227, in http_open
return self.do_open(httplib.HTTPConnection, req)
File "/cvmfs/test.galaxyproject.org/python/lib/python2.7/urllib2.py", line 1197, in do_open
raise URLError(err)
URLError:
```

Jobs are set to the error state with the message "Failed to communicate with remote job server."

Contributor guide

Open the contributing guide

Research direction

The traceback points to galaxy/jobs/runners/pulsar.py:452 in finish_job and the Pulsar client path through full_status and raw_check_complete. Start by tracing the connection-refused case during a RESTful Pulsar restart; done means restarting the remote job server does not incorrectly put jobs into the error state with “Failed to communicate with remote job server.”

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.