apache / apache/fluss

Exclude hadoop dep in fluss-fs-hdfs module

Open
#2,511 3 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
2.1k
Forks
625
Avg merge
3d 14h
Merged PRs (30d)
97

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/fluss/issues) and found nothing similar.

### Motivation

According to the documentation at
https://fluss.apache.org/docs/streaming-lakehouse/integrate-data-lakes/iceberg/#2-remote-storage-support,
when placing fluss-fs-hdfs into FLINK_HOME/lib, if the Hadoop version provided by the environment is not consistent with the one bundled in fluss-fs-hdfs, the Flink job may fail at runtime.

Currently, fluss-fs-hdfs packages the entire Hadoop dependency inside the jar, which makes it tightly coupled to a specific Hadoop version. To avoid classpath conflicts and improve compatibility with the runtime environment, we should remove the embedded Hadoop dependencies from this jar.

Furthermore, to better align with the Hadoop version used in existing clusters, it would be preferable to introduce a Maven profile mechanism that allows users to explicitly specify the Hadoop version to compile against.

### Solution

_No response_

### Anything else?

_No response_

### Willingness to contribute

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the fluss-fs-hdfs module and inspect how its Maven dependencies are packaged into the jar. Review the remote storage documentation and existing cluster Hadoop-version expectations, then define completion as an artifact without embedded Hadoop dependencies plus a Maven profile for selecting the compile-time Hadoop version.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.