apache / apache/hudi

[SUPPORT] Index configuration is not persistent

Open
#13,241 2 comments 0 reactions 0 assignees View on GitHub
area:index
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**Describe the problem you faced**

Initially, I found that there is no data about table index configuration in `hoodie.properties`. So, I tried to check allowance of index type changing during a couple of upserts. I chose simple bucket index, and decided to vary number of buckets. Hudi allowed me to write successfully data in this scenario, which resulted in a corrupted dataset.

**To Reproduce**

1. Create COW table (it's easy to check content of parquet files) with simple bucket index, and set number of buckets to 1.
```sql
CREATE TABLE index_persist (
id int,
dt string
) USING HUDI
TBLPROPERTIES (
'primaryKey' = 'id',
'type' = 'cow',
'preCombineField' = 'dt',
'hoodie.index.type' = 'BUCKET',
'hoodie.bucket.index.num.buckets' = '1'
) LOCATION '/tmp/some-folder';")
```
3. Upsert a bunch of records into the table. Check that we have only 1 parquet file now with all records.
```sql
INSERT INTO index_persist VALUES (1, 0), (2, 0), (3, 0), (4, 0), (5, 0);
```
3. Increase number of buckets, for instance, up to 2.
```sql
SET hoodie.bucket.index.num.buckets=2;
```
5. Upsert records with the same record keys from step 2, but with changed other values. Check that we have 3 parquet files now.
```sql
INSERT INTO index_persist VALUES (1, 100), (2, 100), (3, 100), (4, 100), (5, 100);
```
7. Select all records from the table, and check that we have a mess in records.
```sql
SELECT * FROM index_persist ORDER BY id;
```
Results of select:
> ('1', '', '00000000-0a6e-4a5d-af56-c804f7a69372-0_0-34-54_20250430181205708.parquet', 1, '100')
('2', '', '00000000-0a6e-4a5d-af56-c804f7a69372-0_0-34-54_20250430181205708.parquet', 2, '0')
('2', '', '00000001-62c8-49ba-9298-d0091a20f8e3-0_1-34-55_20250430181205708.parquet', 2, '100')
('3', '', '00000000-0a6e-4a5d-af56-c804f7a69372-0_0-34-54_20250430181205708.parquet', 3, '100')
('4', '', '00000000-0a6e-4a5d-af56-c804f7a69372-0_0-34-54_20250430181205708.parquet', 4, '0')
('4', '', '00000001-62c8-49ba-9298-d0091a20f8e3-0_1-34-55_20250430181205708.parquet', 4, '100')
('5', '', '00000000-0a6e-4a5d-af56-c804f7a69372-0_0-34-54_20250430181205708.parquet', 5, '100')

Corresponding script is available at:
https://github.com/geserdugarov/test-hudi-issues/blob/main/check-index-persistence/check-index-persistence.py

**Expected behavior**

We need to check index configuration, and don't allow to write differently by different writers.

**Environment Description**

* Hudi version : master, commit f0fcbf6eaf39dfe79e2b27ff7d626b0a8c06bce0

* Spark version : 3.5.3

* Storage (HDFS/S3/GCS..) : local file system

* Running on Docker? (yes/no) : no

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the linked check-index-persistence/check-index-persistence.py reproduction and inspect how hoodie.properties and bucket-index settings are handled across the two upserts. Reproduce the duplicate records with Spark 3.5.3, then trace the configuration validation path; done means writers cannot use differing index configurations and the reported upsert no longer corrupts the dataset.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark, sql
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.