box / box/ClusterRunner

Git fetches should be retried on failure

Open
#300 3 comments 0 reactions 0 assignees View on GitHub
enhancement help wanted minor
Dominant language
Python
Stars
184
Forks
41
PR merge metrics
No merged PRs in 30d

Description

Occasionally a git fetch will fail due to network flakiness. It's appropriate to do retries with backoff on this type of failure similar to how we already do retries with direct network requests.

Example failure log is below:

```
[2016-03-22 14:48:46.944] 6008 NOTICE Bld411-Preparat git Command exited with non-zero exit code.
Command: export GIT_ASKPASS="/home/jenkins/.clusterrunner/dist/bin/git_askpass.sh"; export GIT_SSH="/home/jenkins/.clusterrunner/dist/bin/git_ssh.sh"; export PROJECT_DIR="/tmp/clusterrunner_build_symlinks/ff0bffbd-b207-4c83-92d7-42ffad47a4dd"; export GIT_SSH_ARGS="-o BatchMode=yes -o StrictHostKeyChecking=no"; git fetch --update-head-ok origin refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1
Exit code: 128
Console output: fatal: Couldn't find remote ref refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1
fatal: The remote end hung up unexpectedly

[2016-03-22 14:48:46.944] 6008 WARNING Bld411-Preparat build Build 411 failed: Could not fetch specified branch "refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1" from remote "origin". Command: "git fetch --update-head-ok origin refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1"
Output: "fatal: Couldn't find remote ref refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1
fatal: The remote end hung up unexpectedly
"
[2016-03-22 14:48:46.944] 6008 DEBUG Bld411-Preparat build_fsm Build 411 transitioned from PREPARING to ERROR
[2016-03-22 14:48:46.944] 6008 ERROR Bld411-Preparat build_request_h Could not handle build request for build 411.
Traceback (most recent call last):
File "/home/jenkins/ClusterRunnerBuild/app/master/build_request_handler.py", line 103, in _prepare_build_async
build.prepare(self._subjob_calculator)
File "/home/jenkins/ClusterRunnerBuild/app/master/build.py", line 137, in prepare
self.project_type.fetch_project()
File "/home/jenkins/ClusterRunnerBuild/app/project_type/project_type.py", line 105, in fetch_project
self._fetch_project()
File "/home/jenkins/ClusterRunnerBuild/app/project_type/git.py", line 179, in _fetch_project
error_msg='Could not fetch specified branch "{}" from remote "{}".'.format(self._branch, self._remote)
File "/home/jenkins/ClusterRunnerBuild/app/project_type/git.py", line 231, in _execute_git_command_in_repo_and_raise_on_failure
return self._execute_and_raise_on_failure(command, error_msg, cwd=self._repo_directory, env_vars=env_vars)
File "/home/jenkins/ClusterRunnerBuild/app/project_type/project_type.py", line 121, in _execute_and_raise_on_failure
raise RuntimeError('{} Command: "{}"\nOutput: "{}"'.format(message, command, output))
RuntimeError: Could not fetch specified branch "refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1" from remote "origin". Command: "git fetch --update-head-ok origin refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1"
Output: "fatal: Couldn't find remote ref refs/merge-queue/scm/621dc194bfb4e8a923067a8ecb50e2da465ff7d1
fatal: The remote end hung up unexpectedly
"
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with app/project_type/git.py, especially _fetch_project and _execute_git_command_in_repo_and_raise_on_failure, then read app/project_type/project_type.py for the existing network-retry behavior. The change is done when transient git fetch failures are retried with backoff instead of immediately failing the build; use the shown fetch command and failure output to guide verification.

Written by the indexing model from the issue text.

Assessment

Tech stack
git, python
Domain
build-system
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.