box / box/ClusterRunner

Flaky test: test_get_artifact_before_it_is_ready

Open
#276 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Log from TravisCI is below. I was able to reproduce the exact order of log statements by adding sleeps, but was not able to reproduce the failure (the test still passed / the framework didn't need to hard kill the service). We probably want to add some logging to debug this further and figure out why the framework resorted to a hard kill.

(You can tell it was hard killed by the exit code of -9. The functional test framework uses `app.util.process_utils.kill_gracefully`, which tries a terminate followed by a kill.)

```
test_get_artifact_before_it_is_ready (test.functional.master.test_endpoints.TestMasterEndpoints) ... fatal: Needed a single revision
[2015-12-14 00:42:53.446] 1985 DEBUG MasterTornadoTh log
**************************************************
ClusterRunner
* Version: 0.5.???
* PID: 1985
**************************************************
[2015-12-14 00:42:53.456] 1985 INFO MasterTornadoTh subcommand Master service is running on testing-worker-linux-docker-6ffd9a96-3341-linux-16:43000.
[2015-12-14 00:42:53.692] 1985 DEBUG MasterTornadoTh build_fsm Build 1 transitioned from none to QUEUED
[2015-12-14 00:42:53.693] 1985 DEBUG MasterTornadoTh directory Project directory is /tmp
[2015-12-14 00:42:53.693] 1985 INFO MasterTornadoTh event_log Queued request for build 1.
[2015-12-14 00:42:53.693] 1985 INFO RequestHandlerL build_request_h Creating project lock [/tmp] for build 1
[2015-12-14 00:42:53.696] 1985 ERROR MasterTornadoTh cluster_base_ha Exception occurred during request to /v1/build/1/artifacts.tar.gz.
Traceback (most recent call last):
File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/tornado/web.py", line 1346, in _when_complete
callback()
File "/home/travis/virtualenv/python3.4.2/lib/python3.4/site-packages/tornado/web.py", line 1367, in _execute_method
self._when_complete(method(*self.path_args, **self.path_kwargs),
File "/home/travis/build/box/ClusterRunner/app/web_framework/cluster_master_application.py", line 289, in get
artifact_file_path = self._cluster_master.get_path_for_build_results_archive(int(build_id))
File "/home/travis/build/box/ClusterRunner/app/master/cluster_master.py", line 312, in get_path_for_build_results_archive
raise ItemNotReadyError('Build artifact file is not yet ready. Try again later.')
app.util.exceptions.ItemNotReadyError: Build artifact file is not yet ready. Try again later.
[2015-12-14 00:42:53.700] 1985 NOTICE MasterTornadoTh build Request received to cancel build 1.
[2015-12-14 00:42:53.701] 1985 DEBUG MasterTornadoTh build_fsm Build 1 transitioned from QUEUED to CANCELED
[2015-12-14 00:42:53.704] 1985 INFO MainThread unhandled_excep SIGTERM signal received. Triggering teardown.
[2015-12-14 00:42:53.705] 1985 DEBUG MainThread unhandled_excep Executing teardown callback: .remove_pid_file at 0x2b1a766669d8>
[2015-12-14 00:42:53.705] 1985 DEBUG MainThread unhandled_excep Executing teardown callback: functools.partial(>, callback=>)
[2015-12-14 00:42:53.706] 1985 NOTICE MasterTornadoTh subcommand Master server was stopped.
[2015-12-14 00:42:53.707] 1985 INFO Bld1-Preparatio build_request_h Build 1 is waiting for the project lock
FAIL

...

======================================================================
FAIL: test_get_artifact_before_it_is_ready (test.functional.master.test_endpoints.TestMasterEndpoints)
----------------------------------------------------------------------
Traceback (most recent call last):
File "/home/travis/build/box/ClusterRunner/test/framework/functional/base_functional_test_case.py", line 69, in tearDown
service.return_code,
AssertionError: -9 != 0 : Service running on url: http://localhost:43000 should exit with code 0, but exited with code -9.
----------------------------------------------------------------------

```

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.