apache / apache/hudi

[SUPPORT]Flink writes MOR table, both RO table and RT table read nothing by hive

Open
#10,465 15 comments 0 reactions 0 assignees View on GitHub
engine:hive priority:high status:triaged
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

I use Flink write HUDI MOR table, and Flink read table normally, while RO table and RT table read nothing by hive

**To Reproduce**

Steps to reproduce the behavior:

1. Create hive catalog as follows:

CREATE CATALOG hudi_hive_catalog
WITH (
'type'='hudi',
'catalog.path' = 'cosn://bigdata-xxx/user/hive/warehouse',
'hive.conf.dir' = '/usr/local/service/hive/conf',
'mode'='hms',
'table.external' = 'true',
'default-database' = 'hudi_default'
);

2. Use catalog and create table as follows:

use CATALOG hudi_hive_catalog;

CREATE TABLE t1(
mid BIGINT PRIMARY KEY NOT ENFORCED,
uuid VARCHAR(20),
name VARCHAR(10),
age INT,
ts BIGINT,
part INT
)
PARTITIONED BY (part)
WITH (
'connector' = 'hudi',
'path' = 'cosn://bigdata-xxx/user/hive/warehouse/hudi_default.db/t1',
'table.type' = 'MERGE_ON_READ',
'hive_sync.enable' = 'true',
'hive_sync.mode' = 'hms',
'hive_sync.metastore.uris' = 'thrift://xxx:9083'
)

3.Insert some data by Flink
INSERT INTO t1
VALUES
(1,'334e26e9-8355-45cc-97c6-c31daf0df330','nick', 18,1695159649087,20230108),
(2,'334e26e9-8355-45cc-97c6-c31daf0df330','jack', 18,1695159649087,20230109);

4.Read data by Flink and get right records
SELECT * FROM t1;
![image](https://github.com/apache/hudi/assets/12593964/30cb73e2-22a1-414f-8035-b52f5ad9e6ac)

5. Use 'select *' reading data by Hive and get nothing
select * from t1_rt;
![image](https://github.com/apache/hudi/assets/12593964/6ba664d5-f9af-4a80-bd1c-a4b258b1ad30)
select * from t1_ro;
![image](https://github.com/apache/hudi/assets/12593964/da205ace-79a8-4a98-98f2-5d37ccf91c56)

7. Read data with 'order by clause' by Hive and get right results
select * from t1_rt order by mid;
![image](https://github.com/apache/hudi/assets/12593964/0965d50c-f390-43b0-be11-e5e3a21f0d3c)
select * from t1_ro order by mid;
![image](https://github.com/apache/hudi/assets/12593964/eeab5e22-8ef9-4112-8a4e-2389489a2373)

8.Huid files on cos:
![image](https://github.com/apache/hudi/assets/12593964/8871477d-7713-499f-ac47-2b8644f46220)

9.Test COW table is ok

**Expected behavior**
Reading nothing from RO table may be OK, because it only has a log file and without parquet base files,
but reading nothing from RT table is confused, your help is appreciative.

**Environment Description**

* Hudi version :0.14.1

* Spark version :3.2.2

* Hive version :3.1.3

* Hadoop version :3.2.2

* Storage (HDFS/S3/GCS..) :COS on Tencent Cloud

* Running on Docker? (yes/no) :

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the issue with the provided Flink catalog and MERGE_ON_READ table configuration, then compare Hive queries against t1_rt and t1_ro with and without ORDER BY. Check the Hudi, Hive, Spark, and Hadoop versions and COS storage setup shown in the report. Done means both Hive read paths return the records written by Flink, or the expected limitation is documented.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, spark
Domain
data-engineering, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.