apache / apache/hudi

Failed to determine whether the table is a hoodie table

Open
#9,702 6 comments 0 reactions 0 assignees View on GitHub
area:sql priority:medium
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

spark: 3.2.4
hadoop:3.2.4
hudi:0.13.1

create table hudi_cow_nonpcf_tbl (
uuid int,
name string,
price double
) using hudi
tblproperties (
primaryKey = 'uuid'
);

insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20;

spark-sql (default)> insert into hudi_cow_nonpcf_tbl select 1, 'a1', 20;
2023-09-13 11:09:37,256 WARN adapter.Spark3_2Adapter: Failed to determine whether the table is a hoodie table
com.fasterxml.jackson.core.io.JsonEOFException: Unexpected end-of-input: expected close marker for Object (start marker at [Source: (String)"{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}"; line: 1, column: 183])
at [Source: (String)"{"type":"struct","fields":[{"name":"_hoodie_commit_time","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_commit_seqno","type":"string","nullable":true,"metadata":{}},{"name":"_hoodie_record_key","type":"string","nullable":true,"metadata":{}"; line: 1, column: 257]


Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with Spark 3.2.4, Hadoop 3.2.4, and Hudi 0.13.1 using the supplied table-creation and insert commands. Start at the Spark3_2Adapter warning and its Jackson schema parsing path; done means the insert completes without the truncated JSON warning and hoodie-table detection behaves correctly.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.