When slave is working on a build, and is given a subjob for another build, it crashes the master
- Dominant language
- Python
- Stars
- 184
- Forks
- 41
- PR merge metrics
- No merged PRs in 30d
Description
Last night, this occurred. For complicated reasons (https://github.com/box/ClusterRunner/issues/293), a slave ended up being given a subjob for a build, even though it was already working on a subjob for a different build. Bad as that is, the worst part is that this ended up crashing the Cluster-Master.
Cluster master log:
```
^[[0m^[[32m[2016-01-25 20:31:01.391] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 7 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[1m^[[31m[2016-01-25 20:31:01.395] 36436 ERROR Thread-1543305 unhandled_excep Unhandled exception handler caught exception.
Traceback (most recent call last):
File "/home/jenkins/ClusterRunnerBuild/app/util/safe_thread.py", line 18, in run
super().run()
File "/usr/local/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/master/slave.py", line 109, in _async_start_subjob
response = self._network.post_with_digest(execution_url, post_data, Secret.get(), error_on_failure=True)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 60, in post_with_digest
error_on_failure=error_on_failure)
File "/home/jenkins/ClusterRunnerBuild/app/util/decorators.py", line 38, in function_with_retries
return function(*args, **kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 48, in post
return self._request('POST', *args, **kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 125, in _request
format(url, str(resp.status_code), resp.text))
app.util.network._RequestFailedError: Request to http://ip-10-5-8-148.pod.box.net:43001/v1/build/5447/subjob/229 failed with status_code 400 and response "{"error": "Attempted to start subjob 229 for build 5447, but current build id is 5451."}"
^[[0m^[[32m[2016-01-25 20:31:01.397] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 8 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[32m[2016-01-25 20:31:01.401] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 9 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[33m[2016-01-25 20:31:01.421] 36436 NOTICE MasterTornadoTh subcommand Master server was stopped.
```
I haven't investigated carefully enough yet, but this doesn't seem like a situation that would merit shutting down the master.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.