apache / apache/polaris

Forcefully disallow usage of HadoopFileIO

Open
#1,571 2 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
2.1k
Forks
522
Avg merge
1d 22h
Merged PRs (30d)
137

Description

### Describe the bug

`HadoopFileIO`, as the name implies, relies a lot on Hadoop dependencies. Hadoop in turn relies a lot on the Java Security Manager, which is [permanently disabled in Java 24](https://inside.java/2024/12/11/quality-heads-up/). This means that starting with Java 24 `HadoopFileIO` can no longer work.

`HadoopFileIO` security model is also different (Java security manager) and quite more complex than the security model of object stores and not that tightly integrated w/ Polaris anyway.

On top, `HadoopFileIO` is not recommended for Polaris production use.

Considering how long it takes Hadoop to catch up w/ Java releases, it is safer to remove all dependencies, mostly test code requirements, to HadoopFileIO.

### To Reproduce

_No response_

### Actual Behavior

_No response_

### Expected Behavior

_No response_

### Additional context

_No response_

### System information

_No response_

Contributor guide

Open the contributing guide

Research direction

Start by searching the repository for HadoopFileIO and Hadoop dependency declarations, especially the test code requirements mentioned in the issue. Confirm which production or test paths still use it and remove those dependencies or usages; done means the build and tests no longer require HadoopFileIO.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
backend
Issue type
Refactor
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.