apache / apache/paimon

[Bug] throw Hive DDL and paimon schema mismatched excetion when show the table's schema with the timestamp type

Open
#5,450 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
3.4k
Forks
1.4k
Avg merge
1d 11h
Merged PRs (30d)
396

Description

### Search before asking

- [x] I searched in the [issues](https://github.com/apache/paimon/issues) and found nothing similar.

### Paimon version

paimon-1.0.1

### Compute Engine

hive-3.1.2
spark-3.5.1

### Minimal reproduce step

step 1:
spark paimon catalog
```
spark.sql.extensions=org.apache.paimon.spark.extensions.PaimonSparkSessionExtensions
spark.sql.catalog.spark_catalog=org.apache.paimon.spark.SparkGenericCatalog
spark.sql.catalog.spark_catalog.metastore=hive
spark.sql.catalog.spark_catalog.uri=thrift://metastore:9083
spark.sql.catalog.spark_catalog.warehouse=hdfs://nameservice1/user/hive/warehouse
```
create table use the sparksql
``` sql
create table paimondb.test_ts (
name string,
ts timestamp)
USING paimon
TBLPROPERTIES ( 'primary-key' = 'name');
```
step 2:
show the table's schema on hive
```sql
show create table paimondb.test_ts;
```

have no columns info but the exceptions in the log:
```
java.lang.IllegalArgumentException: Hive DDL and paimon schema mismatched! It is recommended not to write any column definition as Paimon external table can read schema from the specified location.
Mismatched fields are:
Field #7
Hive DDL : ts timestamp
Paimon Schema: ts timestamp with local time zone

at org.apache.paimon.hive.HiveSchema.checkFieldsMatched(HiveSchema.java:274) ~[paimon-hive-connector-3.1-1.0.1.jar:1.0.1]
at org.apache.paimon.hive.HiveSchema.extract(HiveSchema.java:165) ~[paimon-hive-connector-3.1-1.0.1.jar:1.0.1]
at org.apache.paimon.hive.PaimonSerDe.initialize(PaimonSerDe.java:67) ~[paimon-hive-connector-3.1-1.0.1.jar:1.0.1]
at org.apache.hadoop.hive.serde2.AbstractSerDe.initialize(AbstractSerDe.java:54) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.serde2.SerDeUtils.initializeSerDeWithoutErrorCheck(SerDeUtils.java:562) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:87) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.metastore.HiveMetaStoreUtils.getDeserializer(HiveMetaStoreUtils.java:77) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializerFromMetaStore(Table.java:289) ~[hive-exec-3.1.2.jar:3.1.2]
at org.apache.hadoop.hive.ql.metadata.Table.getDeserializer(Table.java:282) ~[hive-exec-3.1.2.jar:3.1.2]
```

### What doesn't meet your expectations?

spark create table with the timestamp type, the timestamp type default means 'timestamp with local time zone',and the paimon schema file record the type is 'TIMESTAMP(6) WITH LOCAL TIME ZONE'.
Then, use hive show the table's schema, it throw the exceptions.Hive has supported the 'timestamp with local time zone' type in the vertion 3.1.2, so the error shouldn't ocurr.

### Anything else?

_No response_

### Are you willing to submit a PR?

- [x] I'm willing to submit a PR!

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the issue with the Spark catalog settings and SQL shown, then start in org.apache.paimon.hive.HiveSchema.java at checkFieldsMatched and extract, followed by PaimonSerDe.initialize. Compare the Hive 3.1.2 timestamp representation with the Paimon schema's TIMESTAMP(6) WITH LOCAL TIME ZONE. Done means show create table returns the schema without a Hive DDL and Paimon schema mismatch exception.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.