jenkinsci / jenkinsci/workflow-api-plugin

[JENKINS-61475] Infinite loop in StandardGraphLookupView

Open
#502 2 comments 0 reactions 0 assignees View on GitHub
component:workflow-api-plugin imported-jira-issue pipeline priority:minor resolution:unresolved
Dominant language
Java
Stars
31
Forks
80
Avg merge
1d 16h
Merged PRs (30d)
1

Description

About once a week, multiple threads handling HTTP requests matching the patterns /blue/rest/organizations/jenkins/pipelines//pipelines//runs// or /blue/organizations/jenkins/%2F/detail///pipeline/ will get stuck in an infinite loop in StandardGraphLookupView.bruteForceScanForEnd.

Each thread that get stuck ties a CPU core, and they tend to happen in batches of about 20-30 threads. This results in the Jenkins web UI appearing to be very slow to users. The resolution so far has been to use the JavaMelody monitoring plugin to kill the stuck threads one by one.

I've attached stack traces from two example threads showing that they get stuck in the same place, as well as a snapshot of the CPU usage at the time. There was nothing supicious going on at the time, and I've been unable to manually reproduce the issue.

I believe something like https://github.com/francoiscampbell/workflow-api-plugin/compare/master...graph-lookup-concurrent should address the issue, but I'm not sure how to verify that.

---
Originally reported by fcampbell, imported from: Infinite loop in StandardGraphLookupView


  • status: Open
  • priority: Minor
  • component(s): workflow-api-plugin
  • label(s): pipeline
  • resolution: Unresolved
  • votes: 1
  • watchers: 2
  • imported: 20260601-173816

Raw content of original issue

About once a week, multiple threads handling HTTP requests matching the patterns /blue/rest/organizations/jenkins/pipelines/<folder>/pipelines/<job>/runs/<number>/ or /blue/organizations/jenkins/<folder>%2F<job>/detail/<job>/<number>/pipeline/ will get stuck in an infinite loop in StandardGraphLookupView.bruteForceScanForEnd.

Each thread that get stuck ties a CPU core, and they tend to happen in batches of about 20-30 threads. This results in the Jenkins web UI appearing to be very slow to users. The resolution so far has been to use the JavaMelody monitoring plugin to kill the stuck threads one by one.

I've attached stack traces from two example threads showing that they get stuck in the same place, as well as a snapshot of the CPU usage at the time. There was nothing supicious going on at the time, and I've been unable to manually reproduce the issue.

I believe something like https://github.com/francoiscampbell/workflow-api-plugin/compare/master...graph-lookup-concurrent should address the issue, but I'm not sure how to verify that.

environment

```
OS: Amazon Linux 2

Jenkins version: 2.223

Java version: 8.212.04-r0

JAVA_OPTS: "-Djava.awt.headless=true -XX:ActiveProcessorCount=48 -Dhudson.slaves.NodeProvisioner.initialDelay=0 -Dhudson.slaves.NodeProvisioner.MARGIN=50 -Dhudson.slaves.NodeProvisioner.MARGIN0=0.85"

Installed plugins: in comment

How Jenkins is accessed: Through an AWS Classic Load Balancer

Jenkins installation method: Docker image jenkins/jenkins:2.223-alpine running on ECS, agents launched on-demand by ECS Plugin and connecting via JNLP

Web browser: Various users affected, all using modern versions of Chrome/Firefox

```

4 attachments

- [jenkins-loop-cpu.png](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/50668/jenkins-loop-cpu.png)
> ![jenkins-loop-cpu.png](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/50668/jenkins-loop-cpu.png)
- [jenkins-loop-stack-jelly.txt](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-last/refs/heads/main/attachments/50670/jenkins-loop-stack-jelly.txt)
- [jenkins-loop-stack.txt](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/50671/jenkins-loop-stack.txt)
- [jenkins-loop-threads.png](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/50669/jenkins-loop-threads.png)
> ![jenkins-loop-threads.png](https://raw.githubusercontent.com/jenkinsci/attachments-from-jira-issues-misc/refs/heads/main/attachments/50669/jenkins-loop-threads.png)

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.