adoptium / adoptium/aqa-tests

Limit external tests to startup verification — full functional suites are out of scope for JDK validation

Open
#7,243 5 comments 0 reactions 1 assignee Claimed by @samyoon20 View on GitHub
Dominant language
HTML
Stars
155
Forks
346
Avg merge
2d 9h
Merged PRs (30d)
36

Description

External tests verify Adoptium binaries by running real Java applications (Tomcat, Kafka, Elasticsearch, WildFly, Spring Boot, etc.) inside Docker/Podman containers. Currently every test runs the full upstream test suite of the third-party application — hundreds or thousands of unit/integration tests that can take 30–90+ minutes per application.

This is the wrong tradeoff. Those third-party test suites are designed to find bugs in the application itself. From Adoptium's perspective, a failure in e.g. KafkaProducerTest is almost certainly a Kafka bug or a flaky test environment issue — not a JDK defect. Meanwhile, a JVM crash on startup, a JIT compilation failure, or a class-loading error on the very first java invocation is a JDK defect — and the current approach buries that signal inside hours of unrelated test noise.

This item is also listed in the external/README.md roadmap as a planned improvement: "Startup-only testing (application startup, but not full runs of app functional testing)" — this issue tracks its implementation.

Tests are defined in each project's test.sh. For example https://github.com/adoptium/aqa-tests/blob/master/external/camel/test.sh

to redefine tests in test.sh

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.