Exclude hadoop dep in fluss-fs-hdfs module
- Dominant language
- Java
- Stars
- 2.1k
- Forks
- 625
- Avg merge
- 3d 14h
- Merged PRs (30d)
- 97
Description
### Search before asking
- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.
### Motivation
According to the documentation at
https://fluss.apache.org/docs/streaming-lakehouse/integrate-data-lakes/iceberg/#2-remote-storage-support,
when placing fluss-fs-hdfs into FLINK_HOME/lib, if the Hadoop version provided by the environment is not consistent with the one bundled in fluss-fs-hdfs, the Flink job may fail at runtime.
Currently, fluss-fs-hdfs packages the entire Hadoop dependency inside the jar, which makes it tightly coupled to a specific Hadoop version. To avoid classpath conflicts and improve compatibility with the runtime environment, we should remove the embedded Hadoop dependencies from this jar.
Furthermore, to better align with the Hadoop version used in existing clusters, it would be preferable to introduce a Maven profile mechanism that allows users to explicitly specify the Hadoop version to compile against.
### Solution
_No response_
### Anything else?
_No response_
### Willingness to contribute
- [x] I'm willing to submit a PR!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the fluss-fs-hdfs module and inspect how its Maven dependencies are packaged into the jar. Review the remote storage documentation and existing cluster Hadoop-version expectations, then define completion as an artifact without embedded Hadoop dependencies plus a Maven profile for selecting the compile-time Hadoop version.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system
- Issue type
- Refactor
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 42/100