apache / apache/hudi

Flink bundle does not package Lance runtime dependencies

Closed
#19,864 1 comment 0 reactions 0 assignees View on GitHub
engine:flink type:bug
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Describe the problem

The Hudi Flink bundle does not package the runtime dependencies required by the Lance file format. A Flink job that deploys only `hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar` fails as soon as it initializes the Lance writer.

This was reproduced with a local Flink 2.2.1 standalone cluster and a Hudi table configured with `hoodie.table.base.file.format=LANCE`.

The first missing class is from Arrow:

```text
java.lang.NoClassDefFoundError: org/apache/arrow/memory/BufferAllocator
at org.apache.hudi.io.memory.HoodieArrowAllocator.getRootAllocator(HoodieArrowAllocator.java:67)
at org.apache.hudi.io.memory.HoodieArrowAllocator.newChildAllocator(HoodieArrowAllocator.java:79)
at org.apache.hudi.io.lance.HoodieBaseLanceWriter.(HoodieBaseLanceWriter.java:92)
at org.apache.hudi.io.storage.row.HoodieRowDataLanceWriter.(HoodieRowDataLanceWriter.java:75)
Caused by: java.lang.ClassNotFoundException: org.apache.arrow.memory.BufferAllocator
```

Inspection of the generated bundle confirms that it contains the Hudi Lance reader/writer classes, but not the corresponding `org.apache.arrow` or `org.lance` runtime classes. The Flink bundle shade configuration uses an artifact allowlist that currently omits these dependencies.

The job can proceed only after manually adding the Lance and Arrow runtime jars, including `lance-core`, the Arrow vector/memory/C-data/dataset artifacts, and their required JNI/runtime dependencies, to Flink's `lib` directory.

### Expected behavior

The Hudi Flink bundle should contain the dependencies required to use the Lance base-file format, so that Lance read and write paths work when the bundle is deployed without manually installing additional Lance/Arrow jars.

### Environment

- Flink: 2.2.1
- Hudi: current `1.3.0-SNAPSHOT`
- Deployment: local standalone cluster
- Artifact: `hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar`

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the Flink bundle shade configuration and its artifact allowlist, then inspect hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar for the missing org.apache.arrow and org.lance runtime classes. Reproduce the standalone Flink failure with a table using hoodie.table.base.file.format=LANCE; done means Lance read and write paths work without manually adding runtime jars to Flink's lib directory.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
build-system, stream-processing
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.