apache / apache/paimon

[common] CatalogContext requires Hadoop dependency even in non-Hadoop environments

Open
#6,654 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

## Description

`CatalogContext` currently has a hard dependency on Hadoop Configuration, causing `NoClassDefFoundError` in environments where Hadoop is not needed or available.

## Critical: Trino Plugin Development Blocker

This issue is a **critical blocker** for developing the Trino-Paimon connector.

### Trino's Mandatory Requirement

Trino **explicitly prohibits connectors from having mandatory Hadoop dependencies**:

🔗 **Trino Policy**: https://github.com/trinodb/trino/issues/15921

> **Quote from Trino maintainers:**
> "Trino connectors should not have a hard dependency on Hadoop. Connectors must work without Hadoop on the classpath."

### Previous Paimon-Trino Implementation Issue

The previous `paimon-trino` implementation was affected by this problem:

🔗 **Paimon-Trino Issue**: https://github.com/apache/paimon-trino/issues/96

This issue prevented proper deployment and usage of Paimon with Trino in production environments.

## Use Cases Affected

### 1. 🎯 Trino-Paimon Connector (PRIMARY)
- **Status**: Currently **BLOCKED** by mandatory Hadoop dependency
- **Requirement**: Trino's connector architecture requires optional Hadoop support
- **Impact**: Cannot integrate Paimon with Trino's cloud-native deployment model
- **Policy**: https://github.com/trinodb/trino/issues/15921
- **Previous Issue**: https://github.com/apache/paimon-trino/issues/96

### 2. Windows Development Environment
When using Paimon on Windows (e.g., Flink CDC with Paimon sink to MinIO S3), the application fails with:

```
Caused by: java.lang.NoClassDefFoundError: org/apache/hadoop/conf/Configuration
at org.apache.paimon.catalog.CatalogContext.(CatalogContext.java:53)
at org.apache.paimon.catalog.CatalogContext.create(CatalogContext.java:73)
at org.apache.paimon.flink.FlinkCatalogFactory.createPaimonCatalog(FlinkCatalogFactory.java:81)
at org.apache.flink.cdc.connectors.paimon.sink.v2.bucket.BucketAssignOperator.open(BucketAssignOperator.java:103)
at org.apache.flink.streaming.runtime.tasks.RegularOperatorChain.initializeStateAndOpenOperators(RegularOperatorChain.java:107)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreStateAndGates(StreamTask.java:858)
at org.apache.flink.streaming.runtime.tasks.StreamTask.lambda$restoreInternal$5(StreamTask.java:812)
at org.apache.flink.streaming.runtime.tasks.StreamTaskActionExecutor$1.call(StreamTaskActionExecutor.java:55)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restoreInternal(StreamTask.java:812)
at org.apache.flink.streaming.runtime.tasks.StreamTask.restore(StreamTask.java:771)
at org.apache.flink.runtime.taskmanager.Task.runWithSystemExitMonitoring(Task.java:970)
at org.apache.flink.runtime.taskmanager.Task.restoreAndInvoke(Task.java:939)
at org.apache.flink.runtime.taskmanager.Task.doRun(Task.java:763)
at org.apache.flink.runtime.taskmanager.Task.run(Task.java:575)
at java.base/java.lang.Thread.run(Thread.java:842)
Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.conf.Configuration
```

### 3. Lightweight Deployment Scenarios
- Local FileIO usage shouldn't require Hadoop
- Cloud-native deployments using native S3/OSS clients
- Embedded/lightweight environments
- Container deployments without Hadoop ecosystem

## Problem Root Cause

The current `CatalogContext` implementation has a mandatory Hadoop dependency:

```java
public class CatalogContext {
private final Configuration hadoopConf; // Always requires Hadoop

private CatalogContext(..., Configuration hadoopConf) {
this.hadoopConf = hadoopConf; // Hard dependency - blocks Trino
}

public Configuration hadoopConf() {
return hadoopConf; // Forces ClassNotFoundException without Hadoop
}
}
```

This design:
- ❌ Violates Trino's connector design principles
- ❌ Prevents deployment in Hadoop-free environments
- ❌ Increases dependency footprint unnecessarily
- ❌ Blocks Paimon adoption in cloud-native scenarios

## Proposed Solution

Refactor CatalogContext using class hierarchy to separate concerns:

### 1. `CatalogContext` - Base class (Hadoop-free)
```java
public class CatalogContext {
protected final Options options;
protected final ClassLoader classLoader;
// No Hadoop dependency
}
```

### 2. `HadoopAware` - Interface for Hadoop functionality
```java
public interface HadoopAware {
Configuration hadoopConf();
}
```

### 3. `CatalogHadoopContext` - Hadoop implementation
```java
public class CatalogHadoopContext extends CatalogContext implements HadoopAware {
private final Configuration hadoopConf;

@Override
public Configuration hadoopConf() {
return hadoopConf;
}
}
```

### Components Update
- **FileIO**: Use `CatalogContext` as base type
- **Runtime check**: Components check for `HadoopAware` interface when Hadoop is needed
- **Graceful fallback**: Works without Hadoop, enables Hadoop when available
- **Factory pattern**: Automatically returns appropriate type based on environment

## Expected Benefits

1. ✅ **Unblocks Trino-Paimon connector** - Complies with Trino's requirements
- Resolves https://github.com/trinodb/trino/issues/15921
- Fixes https://github.com/apache/paimon-trino/issues/96
2. ✅ **Fixes Windows development** with Flink CDC + Paimon + MinIO
3. ✅ **Reduces dependency footprint** for cloud-native deployments
4. ✅ **Improves architecture** (separation of concerns)
5. ✅ **Maintains backward compatibility** - existing code continues to work

## Impact Assessment

- **Critical**: Trino connector development (currently blocked)
- **High**: Windows development environments
- **Medium**: Lightweight deployments
- **Low**: Existing Flink/Spark usage (backward compatible)

## Related Issues

- Related to https://github.com/trinodb/trino/issues/15921
- Fixes https://github.com/apache/paimon-trino/issues/96

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reading CatalogContext.java and its create path, then inspect the affected call in FlinkCatalogFactory.java. Trace how the Hadoop Configuration type is loaded and how Hadoop-aware behavior is used. Done means CatalogContext works in Hadoop-free environments while Hadoop-enabled use remains available for the affected Flink CDC path.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
backend, data-engineering
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.