Dependency isolation in Integration Tests
- Dominant language
- Java
- Stars
- 6.1k
- Forks
- 1.5k
- Avg merge
- 1d 21h
- Merged PRs (30d)
- 189
Description
Currently, all of the Pinot integration tests are present in a single module `pinot-integration-tests`. This works fine for most of our modules.
But as we are starting to add more and more integrations such as Pulsar, Kinesis, Pub-Sub etc. the current approach leads to a lot of dependency version conflicts. Most of these conflicts occur in libraries such as netty, servlet-api, guava, jackson etc.
In the individual plugins, these dependencies are generally shaded and hence cause no issues when running in production. Resolving each of these dependencies and finding a correct minor version which is compatible with both existing and new plugin takes some effort.
We may need to come up with a new approach.
Some solutions -
* Add integration tests for plugins in their own respective modules only. This will need us to import integration-test modules and other pinot modules in plugins pom (test scope)
* Find a way to minimize depedency conflicts by reducing the number of imported dependencies.
You can see an example of dependency exclusion because of conflicts in the following draft PR - https://github.com/apache/pinot/pull/8235/files#diff-f191832a3b523ff5a1f818baf6a358e449550c34b75171758749e173dc0b37b2
https://github.com/apache/pinot/pull/8235/files#diff-292e0e1b3c16430f3d1763f6f2aaf13cf30832d51eb7f8ab86b92708135f2013
Contributor guide
Research direction
Start by inspecting the pinot-integration-tests module and the dependency exclusions shown in draft PR #8235. Compare the proposed module-local integration-test approach with reducing imported dependencies, then determine how plugin and integration-test modules should be organized to avoid version conflicts. Done means a documented, agreed approach that resolves the listed dependency conflicts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, testing
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100