jenkinsci / jenkinsci/google-compute-engine-plugin
Workers get blocked by when request several of them in parallel at the same time
- Dominant language
- Java
- Stars
- 58
- Forks
- 97
- PR merge metrics
- No merged PRs in 30d
Description
I'm testing https://github.com/jenkinsci/google-compute-engine-plugin/pull/183 and it seems there are a bunch of threads that get blocked by and therefore the plugin doesn't support a bunch of parallel requests. I also tried with the current latest version and the same behavior.
### Test scenario:
- 300 CAP with a pipeline that runs 100 parallel stages.
- Jenkins core 2.109
- Trilead-api plugin version trilead-api
- Pipeline
Expand to view the Pipeline
```Groovy
import groovy.time.*
pipeline {
agent none
options {
timeout(time: 2, unit: 'HOURS')
timestamps()
disableResume()
durabilityHint('PERFORMANCE_OPTIMIZED')
disableConcurrentBuilds()
}
stages {
stage('Test') {
steps { parallelSteps(100) }
}
}
}
def parallelSteps(num){
def parallelSteps = [:]
for (i = 0; i < num; i++) {
parallelSteps["step-${i}"] = {
echo 'parallelSteps|started'
TimeDuration timeDuration = elapsedTime {
node('gec-linux'){
echo 'hey!'
}
}
echo "parallelSteps|finished|${timeDuration.toString()}"
}
}
parallel parallelSteps
}
def elapsedTime(Closure closure){
def timeStart = new Date()
closure()
def timeStop = new Date()
TimeCategory.minus(timeStop, timeStart)
}
```
- JCasC
Expand to view the JCasC
```
jenkins:
agentProtocols:
- "JNLP4-connect"
- "Ping"
clouds:
- computeEngine:
cloudName: "gce-test"
configurations:
- bootDiskAutoDelete: true
bootDiskSizeGb: 10
bootDiskSourceImageProject: "gce-test"
bootDiskType: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f/diskTypes/pd-ssd"
description: "gce-test workers"
externalAddress: true
javaExecPath: "java"
labelString: "linux immutable gce-test"
labels: "linux immutable gce-test"
launchTimeoutSeconds: 300
machineType: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f/machineTypes/g1-small"
mode: NORMAL
namePrefix: "gce-test-linux"
networkConfiguration:
autofilled:
network: "https://www.googleapis.com/compute/v1/projects/gce-test/global/networks/default"
subnetwork: "https://www.googleapis.com/compute/v1/projects/gce-test/regions/us-central1/subnetworks/default"
numExecutors: 1
oneShot: true
preemptible: true
region: "https://www.googleapis.com/compute/v1/projects/gce-test/regions/us-central1"
retentionTimeMinutes: 6
runAsUser: "jenkins-gce"
serviceAccountEmail: "jenkins-gce@gce-test.iam.gserviceaccount.com"
template: "https://www.googleapis.com/compute/v1/projects/gce-test/global/instanceTemplates/test-ci-ubuntu-1804-lts-a075d8a8"
zone: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f"
credentialsId: "gce-test"
instanceCapStr: "300"
instanceId: "aad4387f-26d5-490f-8455-d8e4c15de6a1"
noDelayProvisioning: true
projectId: "gce-test"
```
I managed to create the below dump analysis from the stacktrace:
- [dump](https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDIvMTAvLS1vdXRwdXQtMi5sb2ctLTEyLTEwLTU5OzstLW91dHB1dC0xLmxvZy0tMTItMTAtNTk7Oy0tb3V0cHV0LmxvZy0tMTItMTAtNTk=)
I did create three different dumps to see how the plugin manages those worker requests.
### Analysis.
- Workers get created as expected.
- The connection is not yet available and gets a timeout.
- The plugin gets a degradation when requests are massive.
### Questions
- How many agents/workers do the plugin can cope to run in parallel without a particular timeout degradation?
- Have you seen this behaviour already?
Contributor guide
Assessment
This issue has not been assessed yet.