Limit external tests to startup verification — full functional suites are out of scope for JDK validation
- Dominant language
- HTML
- Stars
- 155
- Forks
- 346
- Avg merge
- 2d 9h
- Merged PRs (30d)
- 36
Description
External tests verify Adoptium binaries by running real Java applications (Tomcat, Kafka, Elasticsearch, WildFly, Spring Boot, etc.) inside Docker/Podman containers. Currently every test runs the full upstream test suite of the third-party application — hundreds or thousands of unit/integration tests that can take 30–90+ minutes per application.
This is the wrong tradeoff. Those third-party test suites are designed to find bugs in the application itself. From Adoptium's perspective, a failure in e.g. KafkaProducerTest is almost certainly a Kafka bug or a flaky test environment issue — not a JDK defect. Meanwhile, a JVM crash on startup, a JIT compilation failure, or a class-loading error on the very first java invocation is a JDK defect — and the current approach buries that signal inside hours of unrelated test noise.
This item is also listed in the external/README.md roadmap as a planned improvement: "Startup-only testing (application startup, but not full runs of app functional testing)" — this issue tracks its implementation.
Tests are defined in each project's test.sh. For example https://github.com/adoptium/aqa-tests/blob/master/external/camel/test.sh
to redefine tests in test.sh
Contributor guide
Assessment
This issue has not been assessed yet.