Improve integration tests performance and fix the remote Gradle cache
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 92
- Forks
- 81
- Avg merge
- 1d 9h
- Merged PRs (30d)
- 46
Description
Describe the bug
Our current CI configuration forces full rebuilds on every workflow execution, even when running a small and targeted subset of tests (e.g. NodeJS and Python sample service tests). This significantly increases CI execution time and resource usage.
Ideally, build-related tasks (such as Java compilation) should be reused via Gradle up-to-date checks and build cache, and CI workflows should focus on executing only the relevant test suites when no related code has changed.
It appears there are different issues and inconsistencies involved. Firstly, Gradle remote build cache access is unreliable, sometimes failing with 401 or 403 responses when attempting to store/retrieve cache entries:
Could not store entry <cachekey> in remote build cache: Storing entry at 'https://zowe.jfrog.io/zowe/cache/<cachekey>' response status 401:
As a result, CI workflows are often unable to reliably reuse cached build outputs across multiple runs, reducing the effectiveness of the remote build cache.
In addition, many tasks explicitly disable Gradle’s incremental build and caching mechanisms by using:
outputs.upToDateWhen { false }
outputs.cacheIf { false }
This probably causes Gradle to always recompile and always rerun tests and skip any caching.
The Verify that all services can register is the slowest workflow (runStartUpCheck), taking around 20 minutes to be completed. In this case, probably the slowness goes beyond the caching issue.
Additional context
Add any other context about the problem here.
Willingness to help
If you are willing to help with debugging and reproducing the problem or
with fixing the problem, please state your willingness to assist in the created issue.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by inspecting the CI workflows and Gradle task configuration related to targeted NodeJS and Python sample service tests, including the Verify that all services can register workflow and its runStartUpCheck entry point. Reproduce the remote cache 401/403 failures and measure whether removing forced rebuild behavior improves reuse; done means reliable cache access and reduced unnecessary work in the affected workflows.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100