[build] fluss-filesystems uber-jars ship unshaded jackson + guava that can shadow downstream dependencies
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Fluss version
main (development)
### Please describe the bug 🐞
While fixing the same class of shading leak in `fluss-fs-s3` (#4073, follow-up to
#3884), I checked the rest of the `fluss-filesystems` family. I rebuilt every fs
plugin from current `main` and inspected the produced shaded uber-jar. **All
seven ship unshaded `com.fasterxml` (jackson) and `com.google` (guava)** that
will shadow any downstream application's own jackson / guava on the classpath,
causing `NoSuchMethodError` / `NoClassDefFoundError` / silent version skew — the
same failure mode reported in #3884.
### Measured leak (top-level unshaded entries in the shaded uber-jar)
| Plugin | jackson (`com/fasterxml`) | guava (`com/google`) | MRJ escaped* | notes |
| -------------- | ------------------------- | -------------------- | ------------ | ---------------------------------------------------------------------------------- |
| fluss-fs-hdfs | **1,073** | **2,731** | 25 | baseline |
| fluss-fs-s3 | **1,093** | **2,042** | 25 | leaks on `main`; fix proposed in open PR #4073 (not yet merged) |
| fluss-fs-oss | **1,073** | **2,230** | 25 | |
| fluss-fs-cos | **1,073** | **2,246** | 25 | |
| fluss-fs-azure | **1,073** | **2,042** | 25 | |
| fluss-fs-obs | **1,161** | **2,042** | 25 | |
| fluss-fs-gs | **1,073** | **7,351** | 25 | **worst**: ~3.6× the guava of the others, and no relocated copy at all (see below) |
\* MRJ = `META-INF/versions/N/com/fasterxml/jackson/core/io/doubleparser/**`
entries that escape relocation — the exact #3884 bug class, present in all 7.
`fluss-fs-hadoop-shaded` is **not** fine either: as @binary-signal noted in the discussion, it relocates `com.fasterxml` → `org.apache.fluss.fs.shaded.hadoop3.com.fasterxml` correctly and does ship hadoop's guava as `org.apache.hadoop.thirdparty.com.google` (third-partied by hadoop), but it **also** ships its own **unshaded** guava (1,954 `com.google.common`) and commons (2,228 `org.apache.commons`). Those leak into every consumer fs plugin that bundles it — which is part of why the guava counts in the table above run as high as they do. Tracked in #4143, fixed in #4144.
## Root cause (confirmed from pom inspection)
Each fs plugin's own `maven-shade-plugin` config only relocates
`org.apache.commons` → `org.apache.fluss.shaded.org.apache.commons`. It does
**not** relocate `com.fasterxml` or `com.google`. Two consequences:
1. **hdfs / s3 / oss / cos / azure / obs** pull jackson + guava from their direct
deps (hadoop-common, aws-sdk, azure/gcs libs). Those land unshaded at the jar
top level → leak. (These modules also bundle a *second*, correctly-relocated
jackson copy inherited from `fluss-fs-hadoop-shaded`, which is why a relocated
`org/apache/fluss/fs/shaded/hadoop3/com/fasterxml` copy also appears — but the
unshaded copy is what breaks downstream apps.)
2. **fluss-fs-gs** is the worst offender: its shade config relocates nothing
(not even `org.apache.commons`), and it does **not** depend on
`fluss-fs-hadoop-shaded`, so it has no relocated copy to lean on. The Google
Cloud Storage libs drag in a huge `com.google` tree (guava + protobuf + gax),
all unshaded → 7,351 leaked guava entries.
The fix proposed in #4073 (relocate `com.fasterxml` + `com.google` + add the MRJ
exclusion filter) is exactly what the other six need. Note: on `main`, s3 still
leaks like the rest (1,093 jackson + 2,042 guava) — #4073 is an *open PR, not yet
merged*, so s3's fix only lands when that PR merges. Until then, s3 sits in the same
"to-fix" bucket as the other six in this issue; once #4073 merges, s3 drops out and
the remaining six (gs worst) become the work.
## Proposed fix
Two viable strategies:
- **(a) Centralize in `fluss-filesystems` parent pom** — add the relocation block
(`com.fasterxml`, `com.google`, `org.codehaus`, `com.ctc`, `com.google.re2j`,
`org.apache.htrace`) plus the MRJ exclusion filter **once**, so every fs plugin
inherits it. One place to maintain; matches the pattern #4073 used for s3.
This is the lowest-risk way to get all 7 consistent.
- **(b) Per-plugin fix** — mirror #4073 into each plugin's `pom.xml`
(fluss-fs-gs first, then hdfs/oss/cos/azure/obs; s3 is already covered by
#4073). More localized, easier to review per plugin, but repeats config 6×.
Per-plugin verification: `0` unshaded `com/fasterxml` and `com/google` entries
in the produced uber-jar (base path + MRJ), with constant-pool references
rewritten to the shaded path, and hadoop's own
`org.apache.hadoop.thirdparty.com.google` namespace left untouched.
## Status
- fluss-fs-s3: **verified leaked on `main`** (1,093 jackson + 2,042 guava); fix proposed in open PR #4073 (not merged yet)
- fluss-fs-hdfs: **verified leaked** (1,073 + 2,731) ← natural starting point
- fluss-fs-oss / cos / azure / obs: **verified leaked** (~1,073 jackson + ~2,000–2,250 guava)
- fluss-fs-gs: **verified leaked, worst** (1,073 jackson + 7,351 guava)
Happy to drive this (centralized parent-pom fix, or per-plugin starting with gs)
once #4073 lands.
## References
- #3884 — fluss-client / flink uber-jars jackson MRJ leak (shipped)
- #4073 — fluss-fs-s3 jackson + guava relocation (open PR, not yet merged)
- `fluss-fs-hadoop-shaded/pom.xml` — existing correct relocation pattern to copy
### Solution
_No response_
### Are you willing to submit a PR?
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Read fluss-fs-hadoop-shaded/pom.xml and the fluss-filesystems parent pom to compare the existing relocation pattern with each plugin's pom.xml. Use the approach in open PR #4073 as a reference, then rebuild the affected uber-jars and verify zero unshaded com/fasterxml and com/google entries, including MRJ paths, while preserving Hadoop's third-party namespace.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100