Flink bundle does not package Lance runtime dependencies
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
### Describe the problem
The Hudi Flink bundle does not package the runtime dependencies required by the Lance file format. A Flink job that deploys only `hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar` fails as soon as it initializes the Lance writer.
This was reproduced with a local Flink 2.2.1 standalone cluster and a Hudi table configured with `hoodie.table.base.file.format=LANCE`.
The first missing class is from Arrow:
```text
java.lang.NoClassDefFoundError: org/apache/arrow/memory/BufferAllocator
at org.apache.hudi.io.memory.HoodieArrowAllocator.getRootAllocator(HoodieArrowAllocator.java:67)
at org.apache.hudi.io.memory.HoodieArrowAllocator.newChildAllocator(HoodieArrowAllocator.java:79)
at org.apache.hudi.io.lance.HoodieBaseLanceWriter.(HoodieBaseLanceWriter.java:92)
at org.apache.hudi.io.storage.row.HoodieRowDataLanceWriter.(HoodieRowDataLanceWriter.java:75)
Caused by: java.lang.ClassNotFoundException: org.apache.arrow.memory.BufferAllocator
```
Inspection of the generated bundle confirms that it contains the Hudi Lance reader/writer classes, but not the corresponding `org.apache.arrow` or `org.lance` runtime classes. The Flink bundle shade configuration uses an artifact allowlist that currently omits these dependencies.
The job can proceed only after manually adding the Lance and Arrow runtime jars, including `lance-core`, the Arrow vector/memory/C-data/dataset artifacts, and their required JNI/runtime dependencies, to Flink's `lib` directory.
### Expected behavior
The Hudi Flink bundle should contain the dependencies required to use the Lance base-file format, so that Lance read and write paths work when the bundle is deployed without manually installing additional Lance/Arrow jars.
### Environment
- Flink: 2.2.1
- Hudi: current `1.3.0-SNAPSHOT`
- Deployment: local standalone cluster
- Artifact: `hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar`
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the Flink bundle shade configuration and its artifact allowlist, then inspect hudi-flink2.2-bundle-1.3.0-SNAPSHOT.jar for the missing org.apache.arrow and org.lance runtime classes. Reproduce the standalone Flink failure with a table using hoodie.table.base.file.format=LANCE; done means Lance read and write paths work without manually adding runtime jars to Flink's lib directory.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- build-system, stream-processing
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 55/100