adoptium / adoptium/infrastructure
Investigate / clear up exceptions in jenkins system log
- Dominant language
- Python
- Stars
- 96
- Forks
- 106
- Avg merge
- 1d 23h
- Merged PRs (30d)
- 13
Description
A quick search on the recent ones (today and yesterday) can be done with `grep -h '^[^\ ]*Exception' /var/log/jenkins/jenkins.log /var/log/jenkins/jenkins.log.1 | sort | uniq -c`
The two largest culprits at the moment are
- **31 java.io.EOFException: unexpected stream termination** Looks to be consistently following `SEVERE c.m.a.v.r.AzureVMAgentSSHLauncher#launch: Got exception on agent azurebuildagentxxxxxx`
- **62 java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.** Looks consistently following `WARNING hudson.slaves.NodeProvisioner#update: Unexpected exception encountered while provisioning agent azurebuildagent` so both are with the deprecated Azure plugin
- **657 java.lang.IllegalStateException: completed or broken execution** These seem to be exclusively on [a particular job](https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-temurin/493/) from Feb 20 which we can probably delete `WARNING o.j.p.w.cps.CpsStepContext$2#onFailure: Failed to proceed after CpsStepContext[287:build]:Owner[build-scripts/jobs/jdk8u/jdk8u-windows-x64-temurin/493:build-scripts/jobs/jdk8u/jdk8u-windows-x64-temurin #493]`
- **1329 java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.nio.channels.ClosedChannelException.serialVersionUID accessible: module java.base does not "opens java.nio.channels" to unnamed module @6be968ce** Seems Orka related as it's consistently preceded by `INFO o.j.p.cloudstats.CloudStatistics#getIdFor: No support for cloud-stats-plugin by class io.jenkins.plugins.orka.OrkaProvisionedAgent`
- **63 java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms** No constant cause on first look but many are `No node in `READY` state is available to deploy to` with a mention of orka. The others generally have `
WARNING h.i.i.InstallUncaughtExceptionHandler#handleException` as the preceding line
Based on this clearing up things from the [Azure](https://github.com/adoptium/infrastructure/issues/3199) and [Orka](https://github.com/adoptium/infrastructure/issues/2536) provisioning would help with bullets 1, 2, 4 with 3 being one specific job that I can delete and the final one is partially with orka (Noting that as of 1142 today 100% of those were orka)
Contributor guide
Research direction
Start with the provided grep command against /var/log/jenkins/jenkins.log and jenkins.log.1, then review the linked Azure issue 3199 and Orka issue 2536 for the provisioning failures. Check the referenced Jenkins job 493 as the proposed isolated cause. Done means the exceptions are explained and the relevant Azure, Orka, and job-specific causes are cleared up or documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, java
- Domain
- ci-cd, cloud, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100