box / box/ClusterRunner

When slave is working on a build, and is given a subjob for another build, it crashes the master

Open
#294 1 comment 0 reactions 0 assignees View on GitHub
bug minor
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Last night, this occurred. For complicated reasons (https://github.com/box/ClusterRunner/issues/293), a slave ended up being given a subjob for a build, even though it was already working on a subjob for a different build. Bad as that is, the worst part is that this ended up crashing the Cluster-Master.

Cluster master log:

```
^[[0m^[[32m[2016-01-25 20:31:01.391] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 7 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[1m^[[31m[2016-01-25 20:31:01.395] 36436 ERROR Thread-1543305 unhandled_excep Unhandled exception handler caught exception.
Traceback (most recent call last):
File "/home/jenkins/ClusterRunnerBuild/app/util/safe_thread.py", line 18, in run
super().run()
File "/usr/local/lib/python3.4/threading.py", line 868, in run
self._target(*self._args, **self._kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/master/slave.py", line 109, in _async_start_subjob
response = self._network.post_with_digest(execution_url, post_data, Secret.get(), error_on_failure=True)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 60, in post_with_digest
error_on_failure=error_on_failure)
File "/home/jenkins/ClusterRunnerBuild/app/util/decorators.py", line 38, in function_with_retries
return function(*args, **kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 48, in post
return self._request('POST', *args, **kwargs)
File "/home/jenkins/ClusterRunnerBuild/app/util/network.py", line 125, in _request
format(url, str(resp.status_code), resp.text))
app.util.network._RequestFailedError: Request to http://ip-10-5-8-148.pod.box.net:43001/v1/build/5447/subjob/229 failed with status_code 400 and response "{"error": "Attempted to start subjob 229 for build 5447, but current build id is 5451."}"
^[[0m^[[32m[2016-01-25 20:31:01.397] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 8 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[32m[2016-01-25 20:31:01.401] 36436 DEBUG MasterTornadoTh build_scheduler Sending subjob 9 (build 5449) to slave ip-10-5-8-109.pod.box.net:43001.
^[[0m^[[33m[2016-01-25 20:31:01.421] 36436 NOTICE MasterTornadoTh subcommand Master server was stopped.
```

I haven't investigated carefully enough yet, but this doesn't seem like a situation that would merit shutting down the master.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.