jenkinsci / jenkinsci/git-plugin
[JENKINS-59317] Spurious SCM change with a failed git call (e.g. network error)
- Dominant language
- Java
- Stars
- 694
- Forks
- 1.1k
- Avg merge
- 1h 29m
- Merged PRs (30d)
- 3
Description
Our internal build farm's connection to the internet has its flaws, ups and downs, and sometimes during the downs we see gems like this: an effectively failed SCM poll ("Couldn't get remote head revision") is interpreted as a "Changes found" situation, and causes a build "Started by an SCM change", as the (failed) build's Polling Log says:
````
Polling Log
This page captures the polling log that triggered this build.
Started on Sep 7, 2019 6:15:15 AM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
Unexpected ls-remote output line ''
[poll] Couldn't get remote head revision
Done. Took 23 sec
Changes found
````
In this particular situation (outage) as a result, this build fails quickly during checkout of sources with
```
...
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/42ity/fty-outage.git +refs/heads/release/IPM_Infra-1.4:refs/remotes/origin/release/IPM_Infra-1.4" returned status code 128:
stdout:
stderr: fatal: unable to access 'https://github.com/42ity/fty-outage.git/': The requested URL returned error: 503
...
```
which is a reasonable outcome of a network outage (our proxy failed to fetch => 503)... however when the network comes back, the last-tried (green, long ago) revision is the same as the current HEAD, so no need to rebuild, as the branch's Git polling log says:
````
Git Polling Log
Started on Sep 11, 2019 1:59:13 PM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
````
So we end up with with a latest-known state of a branch being "failed", more prominently so on still-tracked old releases that are in fact untouched for months and years (and won't be, unless a need for some hotfix arises).
As a workaround, I have to go to such jobs and manually rebuild them, so they do not pollute our logs and job lists with red lights, and do not trigger alarms we've made.
I believe the undetermined HEAD revision should either not cause a build, or if as a failsafe we want to try one - be remembered as such (null?) and when the network comes back, cause a re-rebuild with a definitive commit, even if same as tried before, so the latest-build status converges to a valid up-to-date verdict.
---
Originally reported by
jimklimov, imported from: Spurious SCM change with a failed git call (e.g. network error)
Raw content of original issue
Our internal build farm's connection to the internet has its flaws, ups and downs, and sometimes during the downs we see gems like this: an effectively failed SCM poll ("Couldn't get remote head revision") is interpreted as a "Changes found" situation, and causes a build "Started by an SCM change", as the (failed) build's Polling Log says:
````
Polling LogThis page captures the polling log that triggered this build.
Started on Sep 7, 2019 6:15:15 AM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
Unexpected ls-remote output line ''
[poll] Couldn't get remote head revision
Done. Took 23 sec
Changes found
````In this particular situation (outage) as a result, this build fails quickly during checkout of sources with
```
...
Caused by: hudson.plugins.git.GitException: Command "git fetch --tags --progress https://github.com/42ity/fty-outage.git +refs/heads/release/IPM_Infra-1.4:refs/remotes/origin/release/IPM_Infra-1.4" returned status code 128:
stdout:
stderr: fatal: unable to access 'https://github.com/42ity/fty-outage.git/': The requested URL returned error: 503
...
```
which is a reasonable outcome of a network outage (our proxy failed to fetch => 503)... however when the network comes back, the last-tried (green, long ago) revision is the same as the current HEAD, so no need to rebuild, as the branch's Git polling log says:
````
Git Polling LogStarted on Sep 11, 2019 1:59:13 PM
Using strategy: Specific revision
[poll] Last Built Revision: Revision dee1deac2b8bc60039ea3cd2907afa708a967486 (release/IPM_Infra-1.4)
using credential 955c217d-8133-41c2-9235-9ce3936ddca6
> git --version # timeout=10
using GIT_ASKPASS to set credentials 42ity-ci GH Token as password
Setting http proxy: thunderbolt.roz.lab.etn.com:8081
> git ls-remote -h https://github.com/42ity/fty-outage.git # timeout=10
````So we end up with with a latest-known state of a branch being "failed", more prominently so on still-tracked old releases that are in fact untouched for months and years (and won't be, unless a need for some hotfix arises).
As a workaround, I have to go to such jobs and manually rebuild them, so they do not pollute our logs and job lists with red lights, and do not trigger alarms we've made.
I believe the undetermined HEAD revision should either not cause a build, or if as a failsafe we want to try one - be remembered as such (null?) and when the network comes back, cause a re-rebuild with a definitive commit, even if same as tried before, so the latest-build status converges to a valid up-to-date verdict.
Contributor guide
Assessment
This issue has not been assessed yet.