apache / apache/fluss

[build] Unshaded commons, netty and jackson leak from the client, connector, lake and filesystem uber-jars

Open
#4,143 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the issues and found nothing similar.

### Fluss version

main (development)

### Please describe the bug 🐞

Existing issues cover part of this problem: #3553 (jackson MRJ entries in `fluss-client` and `fluss-flink-*`), #4072 (jackson in `fluss-fs-s3`) and #4116 (jackson + guava across the `fluss-filesystems` family). This issue covers the leaks that none of those three track.

I built every module from `main` with the two open PRs (#3884, #4073) applied, and scanned all 43 produced jars for third-party classes sitting at their original package path. **12 jars leak.** Seven of them are already tracked by #4072 / #4116. The remaining leaks are below.

#### 1. commons-math3 / commons-lang3 in the client and both connector families

| Jar | unshaded `org/apache/commons` |
| --- | --- |
| `fluss-client` | 1714 |
| `fluss-flink-1.18`, `1.19`, `1.20`, `2.2`, `2.3` | 1714 each |
| `fluss-spark-3.4`, `fluss-spark-3.5` | 1714 each |

Breakdown: commons-math3 1386, commons-lang3 431 (the classes are inherited, hence the identical counts).

`fluss-common` declares both as compile dependencies and Fluss code imports them directly, but `fluss-client/pom.xml` shades everything with no `` block, so they land at their original package and the connector uber-jars inherit them.

This is the same failure #3960 fixed for the S3/GS/Azure filesystem plugins, where Spark hit `NoSuchMethodError` on a shadowed commons-text class. The Spark connectors — the artifacts most likely to meet a conflicting commons — were never covered.

#### 2. commons and jackson in the lake modules

| Jar | leak |
| --- | --- |
| `fluss-lake-iceberg` | commons 1030 (compress 602, lang3 431, pool 57) |
| `fluss-lake-lance` | jackson 1091, commons 519 (codec + lang3) |

Note this is a different set of commons libraries from the client leak above, so it is not the same root cause.

#### 3. netty in four filesystem plugins

`fluss-fs-azure`, `fluss-fs-cos`, `fluss-fs-obs` and `fluss-fs-oss` each ship **1698 unshaded `io.netty` classes**, pulled in by the cloud SDK HTTP layer. #4116 measures jackson and guava in these plugins but not netty.

#### 4. Correction to #4116 regarding `fluss-fs-hadoop-shaded`

#4116 states that `fluss-fs-hadoop-shaded` "itself is fine". That is true for jackson, but the module also ships:

- **1954 unshaded `com.google.common`** (guava)
- **2228 unshaded `org.apache.commons`**

Measured on the module's pre-existing pom, built clean:

```
package leaked mrj relocated
com/fasterxml 0 0 1033
org/apache/commons 2228 0 0 LEAK
com/google/common 1954 0 0 LEAK
```

The jar does also contain 2094 `org/apache/hadoop/thirdparty/com/google` entries, which is presumably what #4116 refers to — but that is Hadoop's own relocated copy sitting alongside a separate unshaded guava, not instead of it. Since every filesystem plugin bundles this module, the leak propagates to all of them.

#### 5. Not fixable: `fluss-lake-lance` arrow and netty

`fluss-lake-lance` also ships 941 unshaded `org.apache.arrow` and 640 unshaded `io.netty`. **These should not be relocated.** The bundled `libarrow_cdata_jni` exports JNI symbols that embed the Java package name:

```
$ nm -gU libarrow_cdata_jni.dylib | grep Java_org_apache_arrow
Java_org_apache_arrow_c_jni_JniWrapper_exportArray
Java_org_apache_arrow_c_jni_JniWrapper_exportSchema
...
```

Renaming the package makes the JVM look for a symbol the library does not export, failing with `UnsatisfiedLinkError` at runtime rather than at build time. The netty in this jar is Arrow's allocator layer (`io.netty.buffer`, `io.netty.util` only) and is tied to those classes. Recording this so nobody "fixes" it later.

### Solution

Relocate each leaking package into the existing `org.apache.fluss.shaded.*` / `org.apache.fluss.fs.shaded..*` namespaces.

**One constraint applies throughout.** The shade plugin rewrites *every* reference matching a `` pattern, including references to classes the jar does not bundle. A broad `org.apache.commons` or `com.google` pattern therefore turns a soft dependency — absent from the jar but resolvable from the surrounding classpath — into a coordinate nothing can ever provide. Concretely, `fluss-client` references commons-codec and commons-logging without bundling them; `fluss-lake-iceberg` references commons-io and commons-lang 2.x; `fluss-fs-hadoop-shaded` references commons-cli, commons-codec, commons-math3, commons-net, `com.google.protobuf` and `com.google.gson`. Relocating those prefixes wholesale breaks the NameNode with:

```
java.lang.NoClassDefFoundError: org/apache/fluss/shaded/org/apache/commons/cli/ParseException
at org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1713)
```

So each pattern must name the packages actually bundled. Two further cases need an exclusion rather than a rename:

- `io.netty.internal.tcnative` — netty's JNI wrapper around OpenSSL, whose native symbols bind to that package name. Exclude it from the netty relocation; it is referenced but not bundled, so leaving it alone also keeps OpenSSL usable when the host supplies it.
- `org.apache.arrow` in `fluss-lake-lance` — see item 5 above.

Also worth noting for anyone working on this: an incremental build reuses already-relocated classes in `target/classes`, so a changed relocation pattern appears to have no effect until `mvn clean` is run.

**Verification.** There is currently no jar shading check anywhere in the repo or CI, which is why these accumulated. A checker that scans built jars for classes at forbidden package paths, confirms each has a relocated counterpart, and reports relocated references that resolve to nothing would catch regressions and could gate CI.

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with fluss-client/pom.xml and the affected lake and filesystem module POMs; run a clean Maven build because incremental output can preserve old relocations. Compare the produced jars with the package counts and exclusions in the issue, including io.netty.internal.tcnative and org.apache.arrow. Done means the listed bundled dependencies are relocated without breaking soft dependencies or JNI-bound packages.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.