jenkinsci / jenkinsci/google-compute-engine-plugin

Workers get blocked by when request several of them in parallel at the same time

Open
#184 1 comment 2 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
58
Forks
97
PR merge metrics
No merged PRs in 30d

Description

I'm testing https://github.com/jenkinsci/google-compute-engine-plugin/pull/183 and it seems there are a bunch of threads that get blocked by and therefore the plugin doesn't support a bunch of parallel requests. I also tried with the current latest version and the same behavior.

### Test scenario:
- 300 CAP with a pipeline that runs 100 parallel stages.
- Jenkins core 2.109
- Trilead-api plugin version trilead-api
- Pipeline

Expand to view the Pipeline

```Groovy
import groovy.time.*
pipeline {
agent none
options {
timeout(time: 2, unit: 'HOURS')
timestamps()
disableResume()
durabilityHint('PERFORMANCE_OPTIMIZED')
disableConcurrentBuilds()
}
stages {
stage('Test') {
steps { parallelSteps(100) }
}
}
}
def parallelSteps(num){
def parallelSteps = [:]
for (i = 0; i < num; i++) {
parallelSteps["step-${i}"] = {
echo 'parallelSteps|started'
TimeDuration timeDuration = elapsedTime {
node('gec-linux'){
echo 'hey!'
}
}
echo "parallelSteps|finished|${timeDuration.toString()}"
}
}
parallel parallelSteps
}

def elapsedTime(Closure closure){
def timeStart = new Date()
closure()
def timeStop = new Date()
TimeCategory.minus(timeStop, timeStart)
}
```

- JCasC

Expand to view the JCasC

```
jenkins:
agentProtocols:
- "JNLP4-connect"
- "Ping"
clouds:
- computeEngine:
cloudName: "gce-test"
configurations:
- bootDiskAutoDelete: true
bootDiskSizeGb: 10
bootDiskSourceImageProject: "gce-test"
bootDiskType: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f/diskTypes/pd-ssd"
description: "gce-test workers"
externalAddress: true
javaExecPath: "java"
labelString: "linux immutable gce-test"
labels: "linux immutable gce-test"
launchTimeoutSeconds: 300
machineType: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f/machineTypes/g1-small"
mode: NORMAL
namePrefix: "gce-test-linux"
networkConfiguration:
autofilled:
network: "https://www.googleapis.com/compute/v1/projects/gce-test/global/networks/default"
subnetwork: "https://www.googleapis.com/compute/v1/projects/gce-test/regions/us-central1/subnetworks/default"
numExecutors: 1
oneShot: true
preemptible: true
region: "https://www.googleapis.com/compute/v1/projects/gce-test/regions/us-central1"
retentionTimeMinutes: 6
runAsUser: "jenkins-gce"
serviceAccountEmail: "jenkins-gce@gce-test.iam.gserviceaccount.com"
template: "https://www.googleapis.com/compute/v1/projects/gce-test/global/instanceTemplates/test-ci-ubuntu-1804-lts-a075d8a8"
zone: "https://www.googleapis.com/compute/v1/projects/gce-test/zones/us-central1-f"
credentialsId: "gce-test"
instanceCapStr: "300"
instanceId: "aad4387f-26d5-490f-8455-d8e4c15de6a1"
noDelayProvisioning: true
projectId: "gce-test"
```

I managed to create the below dump analysis from the stacktrace:
- [dump](https://fastthread.io/my-thread-report.jsp?p=c2hhcmVkLzIwMjAvMDIvMTAvLS1vdXRwdXQtMi5sb2ctLTEyLTEwLTU5OzstLW91dHB1dC0xLmxvZy0tMTItMTAtNTk7Oy0tb3V0cHV0LmxvZy0tMTItMTAtNTk=)

I did create three different dumps to see how the plugin manages those worker requests.

### Analysis.
- Workers get created as expected.
- The connection is not yet available and gets a timeout.
- The plugin gets a degradation when requests are massive.

### Questions
- How many agents/workers do the plugin can cope to run in parallel without a particular timeout degradation?
- Have you seen this behaviour already?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.