jenkinsci / jenkinsci/workflow-durable-task-step-plugin

Agents enter infinite reconnection loop

Open
#609 0 comments 5 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
46
Forks
110
PR merge metrics
No merged PRs in 30d

Description

## Overview

A regression in version [1464.v2d3f5c68f84c](https://github.com/jenkinsci/workflow-durable-task-step-plugin/tree/1464.v2d3f5c68f84c) was discovered, which likely causes Jenkins agents running as Kubernetes pods to enter an infinite reconnection loop. As confirmed by downpinning, the issue does not occur in version `1458.va_2e10a_a_b_7c4d`. This regression was observed after upgrading to Jenkins 2.528.1 in a multi-controller environment and affected multiple controllers simultaneously.

When a Pipeline job executes a shell step (or any durable task) on a Kubernetes pod agent, the build log begins printing these pairs of messages that repeat indefinitely:

```
my-job-123456-2gl5h-m9d88-m98gq is back online
my-job-123456-2gl5h-m9d88-m98gq seems to be removed or offline (java.lang.IllegalStateException: Can't send a remote FilePath to a different remote channel (current=hudson.remoting.Channel@ec30c1d:my-job-123456-2gl5h-m9d88-m98gq, target=hudson.remoting.Channel@3c440f09:my-job-123456-2gl5h-m9d88-m98gq)); will wait for 5 min 0 sec for it to come back online
```

The loop continues for the entire duration of the build.

Those `current` and `target` channels have different object identities (different hex addresses like `@ec30c1d` vs `@3c440f09`), even though they reference the same agent name. This suggests that the plugin might hold a stale reference to a previous channel instance while the agent has established a new channel.

Affected builds that should take 10-15 minutes can run for hours continuously printing these messages. This, in turn, generates enormous (many Gigabytes) logs, sometimes until all the available disk space on the controller's persistent volume is consumed.

## How to reproduce?

The affected environment runs Jenkins controllers as Kubernetes deployments using the Jenkins Helm chart. Each build spawns an ephemeral pod agent via the [Kubernetes plugin](https://github.com/jenkinsci/kubernetes-plugin). The agents connect to the controller using WebSocket. The issue was initially observed on Jenkins 2.528.1 with the standard set of Pipeline plugins.

The issue is reproducible by upgrading workflow-durable-task-step from version 1458 to version 1464 and running any Pipeline job that executes an `sh` step on a Kubernetes pod agent. Not every build is affected, but on a busy controller running hundreds of builds per day, affected builds appear within hours of the upgrade.

It has not been possible to determine a precise trigger that causes some builds to enter the loop while others complete normally. The affected builds do not share any obvious characteristics in terms of the Jenkinsfile content, repository size, or build duration. The issue appears to be a race condition related to channel lifecycle management.

## Suspected cause

It appears that the regression was introduced in https://github.com/jenkinsci/workflow-durable-task-step-plugin/commit/83386f485ce6730e36b7d8b0569fd837bbd2d083, which modified the `AgentReconnectionListener` class. The previous implementation used `StepExecution.acceptAll` to iterate over all executions when handling agent online/offline events:

https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/bf7088bc3be500e206f750b271303400953ba03b/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L783-L791

The new implementation changed this to iterate over the computer's executors and inspect their flow executions directly:

https://github.com/jenkinsci/workflow-durable-task-step-plugin/blob/83386f485ce6730e36b7d8b0569fd837bbd2d083/src/main/java/org/jenkinsci/plugins/workflow/steps/durable_task/DurableTaskStep.java#L790-L819

Under certain timing conditions in a Kubernetes environment where pods are created and destroyed frequently, the new iteration logic might either trigger checks against stale channel references or cause the `Execution#check` method to be invoked in a context where the channel has already been replaced. The `FilePath` object serialized with the old channel reference then fails validation when sent over the new channel, producing the exception.

## Workaround

Downgrading the plugin to version `1458.va_2e10a_a_b_7c4d` immediately resolves the issue. This workaround has been validated over an extended period with no recurrence of the reconnection loops.

Any investigation into whether the hypothesis listed above is correct and whether changes in

- #478

introduced this regression would be appreciated.

Happy to provide additional diagnostic information, test proposed fixes, or assist with reproduction if needed.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.