apache / apache/flink-agents

[Feature] Shade and relocate third-party dependencies in flink-agents-dist to avoid classpath conflicts

Open
#652 1 comment 0 reactions 1 assignee Claimed by @ysymi View on GitHub
feature fixVersion/0.4.0 priority/major
Dominant language
Java
Stars
452
Forks
167
Avg merge
5d 9h
Merged PRs (30d)
49

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/flink-agents/issues) and found nothing similar.

### Description

## Motivation

The current `flink-agents-dist-*.jar` bundles a large number of third-party
dependencies without relocating them:

- `jackson-databind` 2.18.2
- `kafka-clients` 4.0.0
- `kotlin-stdlib` 1.x (transitive from `openai-java`)
- `anthropic-java`, `openai-java`, etc.

When users drop this fat jar into the Flink cluster's `lib/` directory, or
include it via `--jars` at job submission time, these classes land in the
same classloader as the user's job code. If the user's job also depends on
e.g. `kafka-clients` (a very common scenario in Flink streaming jobs) or
`jackson-databind` with a different version, runtime errors like
`NoSuchMethodError` or `ClassCastException` are likely.

Note that Flink itself already relocates its own bundled dependencies
(e.g. `flink-shaded-jackson`, `flink-shaded-guava`) precisely to avoid
this problem. flink-agents-dist should follow the same approach.

## Analysis

The top offenders by size and conflict risk:

| Dependency | Size | Risk | Notes |
|---|---|---|---|
| `kafka-clients` 4.0.0 | ~22 MB | **High** | Many Flink jobs use Kafka; major version gap |
| `openai-java` + `kotlin-stdlib` | ~80 MB | Medium | Uncommon in user code but large |
| `jackson-databind` 2.18.2 | ~6 MB | Medium | Flink uses shaded jackson, user code may not |
| `anthropic-java` | ~17 MB | Low | Rarely used alongside |

## Proposed Solution

Relocate in shade plugin

Add `` rules in `dist/pom.xml` to move third-party packages
under an `org.apache.flink.agents.shaded.*` namespace:

```xml


com.fasterxml.jackson
org.apache.flink.agents.shaded.jackson


org.apache.kafka
org.apache.flink.agents.shaded.kafka

```

This requires verifying that internal usages of these libraries (especially
Jackson annotations on public API classes, Kafka SPI configurations) still
work correctly after relocation.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.