apache / apache/paimon

[Feature] The paimon table serializationLibrary value does not depend on the paimon jar by default

Open
#2,625 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [X] I searched in the [issues](https://github.com/apache/incubator-paimon/issues) and found nothing similar.

### Motivation

If you run describe extended tableName ddl to obtain table metadata for tables created by spark sql, the following exception occurs. Do you not add PaimonSerDe by default? In the enterprise, hadoop is the commercial version (Huawei mrs, CDP), and complex processes are required to modify the jar that hms relies on. It is not recommended to add PaimonSerDe by default. iceberg and hudi do not have this problem.
```
Caused by: java.sql.SQLException: Error while processing statement: FAILED: Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.DDLTask. java.lang.ClassNotFoundException Class org.apache.paimon.hive.PaimonSerDe not found
```
Hive table properties storage_handler = org.apache.paimon.hive.PaimonStorageHandler might have similar problem

@tsreaper @JingsongLi

hudi serializationLibrary Default value: ParquetHiveSerDe.class.getName()
![image](https://github.com/apache/incubator-paimon/assets/1145830/3c35907a-2868-4f16-af17-7e50e9603e8d)

### Solution

_No response_

### Anything else?

_No response_

### Are you willing to submit a PR?

- [ ] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by tracing the Hive path used by `describe extended tableName` and the handling of `serializationLibrary`, `PaimonSerDe`, and `PaimonStorageHandler`. No source file or test is named, and the expected default is not specified beyond avoiding a required Paimon jar; done should include a decided default and regression coverage for Spark SQL-created tables.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java, spark
Domain
data-engineering, databases
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
32/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.