jenkinsci / jenkinsci/openstack-cloud-plugin

Instance cap is reached although the instances are not allocated in OS

Open
#74 5 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
49
Forks
97
Avg merge
13h 23m
Merged PRs (30d)
6

Description

It seems that either the instance cap is not reset properly, or the message printed to the log is misleading. The following log excerpt corresponds to the state when there were no nodes provisioned by OpenStackCloud plugin visible in QEOS. However, it starts with complaining about reaching the instance cap:

```
May 18, 2016 6:59:04 AM jenkins.plugins.openstack.compute.JCloudsCloud getAvailableTemplateProvider
INFO: Global instance cap (60) reached while adding capacity for label: RHEL
May 18, 2016 6:59:04 AM jenkins.plugins.openstack.compute.JCloudsSlaveTemplate provision
INFO: Provisioning new openstack node os-rhel7-715 with options jenkins.plugins.openstack.compute.SlaveOptions@44b9cef5[imageId=bxms-rhel7.2-snapshot,hardwareId=4,networkId=,userDataId=jenkins.plugins.openstack.compute.UserDataConfig.1454073256508,instanceCap=60,floatingIpPool=public,securityGroups=default,availabilityZone=nova,startTimeout=1800000,keyPairName=bxms-qe-jenkins,numExecutors=1,jvmOptions=-Xmx700m -XX:MaxPermSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=/tmp -Djavax.net.ssl.trustStore=/qa/tools/keystore/mw.truststore -Djavax.net.ssl.trustStorePassword=changeit -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9110 -Dcom.sun.management.jmxremote.authenticate=false -Dcom.sun.management.jmxremote.ssl=false -Dcom.sun.management.jmxremote.local.only=false,fsRoot=/home/hudson/hudson_workspace,credentialsId=c7c5fd01-0bc7-48dc-bb50-c0e46d2b2464,slaveType=SSH,retentionTime=30]
...
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:9.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:8.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:7.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:6.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:5.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:4.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:3.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:2.738923
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:1.7389231
May 18, 2016 6:59:04 AM hudson.slaves.NodeProvisioner update
INFO: Started provisioning os-rhel7 from qeos with 1 executors. Remaining excess workload:0.7389231
```

I still have from time to time clean the OpenStack instances - they are usually not in ERROR state, but in ACTIVE, although they often do not have IP address associated (not even the internal IP, i.e. not the Floating-IP address). I then have to shut them down using QEOS API - could this mean that the plugin is still tracking them somewhere and believes that the instance count is higher than it actually is?

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.