apache / apache/hudi

[SUPPORT] Flink engine support RECORD_INDEX

Open
#11,676 8 comments 0 reactions 0 assignees View on GitHub
area:index type:feature
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

**_Tips before filing an issue_**

- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
-yes

- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- yes

- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.

**Describe the problem you faced**
Currently, the Flink engine does not support RECORD_INDEX. We are using the Flink engine for writing and the Spark engine for querying, requiring extremely high point differential efficiency. Is there any other way to add RECORD_INDEX to this table, or is it possible to enhance the Flink engine to support RECORD_INDEX? If so, what are the steps to submit this feature? This is my first time contributing code to the community. Thank you.

A clear and concise description of the problem.

**To Reproduce**

Steps to reproduce the behavior:

1.
2.
3.
4.

**Expected behavior**

A clear and concise description of what you expected to happen.

**Environment Description**

* Hudi version : 0.15.1

* Spark version : 3.3.1
* Flink Version: 1.17.1

* Hive version : 3.x

* Hadoop version : 3.1.3

* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no

**Additional context**

Add any other context about the problem here.

**Stacktrace**

```Add the stacktrace of the error.```

Contributor guide

No contributing guide indexed for this repository

Research direction

The issue identifies the Flink engine and RECORD_INDEX but does not name files, tests, or an entry point. Start by locating the Flink engine's RECORD_INDEX handling and related Spark behavior, then define how support should be verified for Hudi 0.15.1-compatible environments.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, java
Domain
data-engineering, stream-processing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.