apache / apache/hudi

ClassNotFoundException: MergeOnReadInputSplit

Open
#9,474 1 comment 0 reactions 0 assignees View on GitHub
engine:flink issue:class-conflict
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

### Environment

- Flink:1.17.1
- hudi:1.14.0-rc
- hadoop:3.2.2

### init.sql script
`
SET 'state.checkpoints.dir' = 'hdfs:///hudi/checkpoints/';
SET 'execution.checkpointing.interval' = '20s';
SET 'execution.checkpointing.min-pause' = '5s';
SET 'execution.checkpointing.max-concurrent-checkpoints' = '1';

add jar '/export/server/flink-1.17.1/hudi-flink1.17-bundle-0.14.0-rc1.jar;

create table t_hudi_user(
id BIGINT,
name STRING,
age INT,
sex BOOLEAN,
city String,
birth timestamp(3)
)
PARTITIONED BY (birth)
WITH
(
'connector' = 'hudi',
'hoodie.datasource.write.recordkey.field' = 'id',
'path' = 'hdfs://CentOS:9000/hudi/t_hudi_user',
'table.type' = 'MERGE_ON_READ',
'compaction.trigger.strategy' = 'num_or_time',
'compaction.delta_commits' = '3',
'compaction.delta_seconds' = '300',
'hoodie.datasource.write.hive_style_partitioning' = 'true',
'write.datetime.partitioning'='true',
'write.partition.format'='yyyy-MM-dd',
'hive_sync.assume_date_partitioning' = 'true',
'hive_sync.mode' = 'hms',
'write.precombine.field' = 'birth',
'changelog.enabled' = 'true',
'read.streaming.enabled' = 'true',
'read.streaming.check-interval' = '3',
'compaction.tasks' = '2',
'hive_sync.enable' = 'true',
'hive_sync.table' = 't_hudi_user',
'hive_sync.db' = 'default',
'hive_sync.metastore.uris' = 'thrift://192.168.42.129:9083',
'hoodie.datasource.hive_sync.support_timestamp' = 'true'
);
`
when i execute command as follow,the console termnal raise such error `java.lang.ClassNotFoundException: org.apache.hudi.table.format.mor.MergeOnReadInputSplit`.I'm sure the `MergeOnReadInputSplit` class has already complied in ·hudi-flink1.17-bundle-0.14.0-rc1.jar` jar file.
`
![image](https://github.com/apache/hudi/assets/23492991/33062891-da14-40cd-b591-2c37575a129f)
![image](https://github.com/apache/hudi/assets/23492991/d00114af-aaae-4212-a7b1-a96d64a940f6)

But inserting is okay, querying is not, which makes me feel very strange!!
![image](https://github.com/apache/hudi/assets/23492991/f5d506ae-47d9-4cef-b5dc-a4d55c68c0d5)

I had try a lower flink verison 1.14.x has no tihs problem

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the init.sql reproduction and the hudi-flink1.17-bundle-0.14.0-rc1.jar, then compare class loading during the successful insert and failing query on Flink 1.17. Check the corresponding Flink 1.14.x setup as a comparison. Done means querying the MERGE_ON_READ table no longer raises ClassNotFoundException for MergeOnReadInputSplit.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
stream-processing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.