apache / apache/hudi

[SUPPORT] Trino can't read tables created by Flink Hudi conector

Open
#9,435 23 comments 0 reactions 0 assignees View on GitHub
component:catalog-sync component:trino
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am creating a hudi table using Flink Hudi connector

```sql
CREATE TABLE flink.flink_hudi_hms3 (
uuid VARCHAR(20),
name VARCHAR(10),
age INT,
ts TIMESTAMP(3),
`partition` VARCHAR(20)
)
PARTITIONED BY (`partition`)
WITH (
'connector' = 'hudi',
'path' = 'abfs://flink@test.dfs.core.windows.net/hudi/t1hms3',
'table.type' = 'COPY_ON_WRITE',
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.metastore.uris' = 'thrift://hive-metastore:9083'
);

```
I Insert data into the table

```sql
INSERT INTO flink_hudi_hms3 VALUES
('id1','Danny',23,TIMESTAMP '1970-01-01 00:00:01','par1'),
('id2','Stephen',33,TIMESTAMP '1970-01-01 00:00:02','par1'),
('id3','Julian',53,TIMESTAMP '1970-01-01 00:00:03','par2'),
('id4','Fabian',31,TIMESTAMP '1970-01-01 00:00:04','par2'),
('id5','Sophia',18,TIMESTAMP '1970-01-01 00:00:05','par3'),
('id6','Emma',20,TIMESTAMP '1970-01-01 00:00:06','par3'),
('id7','Bob',44,TIMESTAMP '1970-01-01 00:00:07','par4'),
('id8','Han',56,TIMESTAMP '1970-01-01 00:00:08','par4');
```

I can see the registered table in Trino, but when I try to read data from the table, I get the following error

```
io.trino.spi.TrinoException: line 1:8: SELECT * not allowed from relation that has no columns
at io.trino.sql.analyzer.SemanticExceptions.semanticException(SemanticExceptions.java:48)
at io.trino.sql.analyzer.SemanticExceptions.semanticException(SemanticExceptions.java:43)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.analyzeSelectAllColumns(StatementAnalyzer.java:4249)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.analyzeSelect(StatementAnalyzer.java:4185)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:2913)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuerySpecification(StatementAnalyzer.java:479)
at io.trino.sql.tree.QuerySpecification.accept(QuerySpecification.java:155)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:496)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:504)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:1470)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.visitQuery(StatementAnalyzer.java:479)
at io.trino.sql.tree.Query.accept(Query.java:107)
at io.trino.sql.tree.AstVisitor.process(AstVisitor.java:27)
at io.trino.sql.analyzer.StatementAnalyzer$Visitor.process(StatementAnalyzer.java:496)
at io.trino.sql.analyzer.StatementAnalyzer.analyze(StatementAnalyzer.java:458)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:79)
at io.trino.sql.analyzer.Analyzer.analyze(Analyzer.java:71)
at io.trino.execution.SqlQueryExecution.analyze(SqlQueryExecution.java:268)
at io.trino.execution.SqlQueryExecution.(SqlQueryExecution.java:203)
at io.trino.execution.SqlQueryExecution$SqlQueryExecutionFactory.createQueryExecution(SqlQueryExecution.java:850)
at io.trino.dispatcher.LocalDispatchQueryFactory.lambda$createDispatchQuery$0(LocalDispatchQueryFactory.java:149)
at io.trino.$gen.Trino_0_410_0_1_0_5_2____20230812_012523_2.call(Unknown Source)
at com.google.common.util.concurrent.TrustedListenableFutureTask$TrustedFutureInterruptibleTask.runInterruptibly(TrustedListenableFutureTask.java:131)
at com.google.common.util.concurrent.InterruptibleTask.run(InterruptibleTask.java:74)
at com.google.common.util.concurrent.TrustedListenableFutureTask.run(TrustedListenableFutureTask.java:82)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.base/java.lang.Thread.run(Unknown Source)
```

Show create table in Trino returns no details on the table.

```
trino:flink> show create table flink_hudi_hms3;
Create Table
-------------------------------------------
CREATE TABLE hive.flink.flink_hudi_hms3 (

)
(1 row)
```

What am I missing here? I tried reading this from Trino Hive catalog above, then I use Hudi connector - I get the error
```
USE
trino:flink> select * from flink_hudi_hms3;
Query 20230812_014715_00026_st296 failed: Not a Hudi table: flink.flink_hudi_hms3
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Begin with the Flink Hudi connector table definition and Hive metastore registration shown in the report; compare them with Trino's SHOW CREATE TABLE output. Reproduce SELECT * through the Trino Hive catalog and the Hudi connector, then verify that the registered table exposes the declared columns and can be read without the reported errors.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, sql
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.