apache / apache/pinot

Dependency isolation in Integration Tests

Open
#8,234 6 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
6.1k
Forks
1.5k
Avg merge
1d 21h
Merged PRs (30d)
189

Description

Currently, all of the Pinot integration tests are present in a single module `pinot-integration-tests`. This works fine for most of our modules.

But as we are starting to add more and more integrations such as Pulsar, Kinesis, Pub-Sub etc. the current approach leads to a lot of dependency version conflicts. Most of these conflicts occur in libraries such as netty, servlet-api, guava, jackson etc.

In the individual plugins, these dependencies are generally shaded and hence cause no issues when running in production. Resolving each of these dependencies and finding a correct minor version which is compatible with both existing and new plugin takes some effort.

We may need to come up with a new approach.

Some solutions -

* Add integration tests for plugins in their own respective modules only. This will need us to import integration-test modules and other pinot modules in plugins pom (test scope)

* Find a way to minimize depedency conflicts by reducing the number of imported dependencies.

You can see an example of dependency exclusion because of conflicts in the following draft PR - https://github.com/apache/pinot/pull/8235/files#diff-f191832a3b523ff5a1f818baf6a358e449550c34b75171758749e173dc0b37b2

https://github.com/apache/pinot/pull/8235/files#diff-292e0e1b3c16430f3d1763f6f2aaf13cf30832d51eb7f8ab86b92708135f2013

Contributor guide

Open the contributing guide

Research direction

Start by inspecting the pinot-integration-tests module and the dependency exclusions shown in draft PR #8235. Compare the proposed module-local integration-test approach with reducing imported dependencies, then determine how plugin and integration-test modules should be organized to avoid version conflicts. Done means a documented, agreed approach that resolves the listed dependency conflicts.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.