Discussion: Dynamic nodes can unallocate mid-job

**Summary**

In build and test jobs, Dynamic nodes (Azure and GHA) have been seen deactivating themselves mid-jenkins job.

The current theory is that they are deallocating themselves, removing a job's host from the machine pool entirely.

This issue is to:
- Document the issue
- List causes
- Discuss solutions

**Glossary**

- GHA: GitHub Action (node). These machines are the ones GitHub Actions run on.
- Azure: Azure is a provider that allocates dynamic Linux nodes.
- Dynamic node: A computer (or a slice of one) that is dynamically allocated on request.

**Links**

- Example GHA failure: https://ci.adoptium.net/job/Test_openjdk8_hs_extended.functional_x86-64_mac/693/
- Example Azure failure: https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk26u/job/jdk26u-alpine-linux-x64-temurin/12/consoleFull

**GHA Node Facts**

- GHA dynamic nodes were added to the pool in April 2026
- They are usually Mac nodes.
- They are active for up to 6 hours after creation, and then they shut down.
- These failures may have been solved by making these nodes deallocate immediately after their first job.
- Note: This will only solve things if the job takes less than 6 hours to run.

**Azure nodes**
- Azure dynamic nodes are usually Linux.
- They were originally configured as "Spot" nodes, where Azure can deallocate them at any time.
- The build nodes may not be "Spot" nodes as of June 23rd, 2026.

**Solutions**
- One option is to automatically rerun a build or test job if it fails, especially during releases.
- Pros: Faster releases, as we avoid being blocked by a one-off dynamic node disconnection.
- Cons: We need to ensure that automated reruns do not override the status of the original job.

**Statistics**

Out of 166 builds on azure dynamic nodes, 6 were found to have caused job failure due to this symptom:

- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-x64-temurin/486/
- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk17u/job/jdk17u-alpine-linux-x64-temurin/480/
- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk21u/job/jdk21u-linux-x64-temurin/443/
- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk25u/job/jdk25u-alpine-linux-x64-temurin/53/
- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk25u/job/jdk25u-linux-x64-temurin/73/
- https://ci.adoptium.net/job/build-scripts/job/jobs/job/jdk25u/job/jdk25u-alpine-linux-x64-temurin/40/

Therefore the failure rate is approximately 3.6%