[Feature] Shade and relocate third-party dependencies in flink-agents-dist to avoid classpath conflicts
- Dominant language
- Java
- Stars
- 452
- Forks
- 167
- Avg merge
- 5d 9h
- Merged PRs (30d)
- 49
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.
### Description
## Motivation
The current `flink-agents-dist-*.jar` bundles a large number of third-party
dependencies without relocating them:
- `jackson-databind` 2.18.2
- `kafka-clients` 4.0.0
- `kotlin-stdlib` 1.x (transitive from `openai-java`)
- `anthropic-java`, `openai-java`, etc.
When users drop this fat jar into the Flink cluster's `lib/` directory, or
include it via `--jars` at job submission time, these classes land in the
same classloader as the user's job code. If the user's job also depends on
e.g. `kafka-clients` (a very common scenario in Flink streaming jobs) or
`jackson-databind` with a different version, runtime errors like
`NoSuchMethodError` or `ClassCastException` are likely.
Note that Flink itself already relocates its own bundled dependencies
(e.g. `flink-shaded-jackson`, `flink-shaded-guava`) precisely to avoid
this problem. flink-agents-dist should follow the same approach.
## Analysis
The top offenders by size and conflict risk:
| Dependency | Size | Risk | Notes |
|---|---|---|---|
| `kafka-clients` 4.0.0 | ~22 MB | **High** | Many Flink jobs use Kafka; major version gap |
| `openai-java` + `kotlin-stdlib` | ~80 MB | Medium | Uncommon in user code but large |
| `jackson-databind` 2.18.2 | ~6 MB | Medium | Flink uses shaded jackson, user code may not |
| `anthropic-java` | ~17 MB | Low | Rarely used alongside |
## Proposed Solution
Relocate in shade plugin
Add `` rules in `dist/pom.xml` to move third-party packages
under an `org.apache.flink.agents.shaded.*` namespace:
```xml
com.fasterxml.jackson
org.apache.flink.agents.shaded.jackson
org.apache.kafka
org.apache.flink.agents.shaded.kafka
```
This requires verifying that internal usages of these libraries (especially
Jackson annotations on public API classes, Kafka SPI configurations) still
work correctly after relocation.
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
Assessment
This issue has not been assessed yet.