adoptium / adoptium/infrastructure

Investigate / clear up exceptions in jenkins system log

Open
#3,514 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
96
Forks
106
Avg merge
1d 23h
Merged PRs (30d)
13

Description

A quick search on the recent ones (today and yesterday) can be done with `grep -h '^[^\ ]*Exception' /var/log/jenkins/jenkins.log /var/log/jenkins/jenkins.log.1 | sort | uniq -c`

The two largest culprits at the moment are

- **31 java.io.EOFException: unexpected stream termination** Looks to be consistently following `SEVERE c.m.a.v.r.AzureVMAgentSSHLauncher#launch: Got exception on agent azurebuildagentxxxxxx`
- **62 java.io.IOException: Agent failed to connect, even though the launcher didn't report it. See the log output for details.** Looks consistently following `WARNING hudson.slaves.NodeProvisioner#update: Unexpected exception encountered while provisioning agent azurebuildagent` so both are with the deprecated Azure plugin
- **657 java.lang.IllegalStateException: completed or broken execution** These seem to be exclusively on [a particular job](https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk8u/job/jdk8u-windows-x64-temurin/493/) from Feb 20 which we can probably delete `WARNING o.j.p.w.cps.CpsStepContext$2#onFailure: Failed to proceed after CpsStepContext[287:build]:Owner[build-scripts/jobs/jdk8u/jdk8u-windows-x64-temurin/493:build-scripts/jobs/jdk8u/jdk8u-windows-x64-temurin #493]`
- **1329 java.lang.reflect.InaccessibleObjectException: Unable to make field private static final long java.nio.channels.ClosedChannelException.serialVersionUID accessible: module java.base does not "opens java.nio.channels" to unnamed module @6be968ce** Seems Orka related as it's consistently preceded by `INFO o.j.p.cloudstats.CloudStatistics#getIdFor: No support for cloud-stats-plugin by class io.jenkins.plugins.orka.OrkaProvisionedAgent`
- **63 java.util.concurrent.TimeoutException: Idle timeout expired: 30000/30000 ms** No constant cause on first look but many are `No node in `READY` state is available to deploy to` with a mention of orka. The others generally have `
WARNING h.i.i.InstallUncaughtExceptionHandler#handleException` as the preceding line

Based on this clearing up things from the [Azure](https://github.com/adoptium/infrastructure/issues/3199) and [Orka](https://github.com/adoptium/infrastructure/issues/2536) provisioning would help with bullets 1, 2, 4 with 3 being one specific job that I can delete and the final one is partially with orka (Noting that as of 1142 today 100% of those were orka)

Contributor guide

Open the contributing guide

Research direction

Start with the provided grep command against /var/log/jenkins/jenkins.log and jenkins.log.1, then review the linked Azure issue 3199 and Orka issue 2536 for the provisioning failures. Check the referenced Jenkins job 493 as the proposed isolated cause. Done means the exceptions are explained and the relevant Azure, Orka, and job-specific causes are cleared up or documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
ci-cd, cloud, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.