[SUPPORT] Flink engine support RECORD_INDEX
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
**_Tips before filing an issue_**
- Have you gone through our [FAQs](https://hudi.apache.org/learn/faq/)?
-yes
- Join the mailing list to engage in conversations and get faster support at dev-subscribe@hudi.apache.org.
- yes
- If you have triaged this as a bug, then file an [issue](https://issues.apache.org/jira/projects/HUDI/issues) directly.
**Describe the problem you faced**
Currently, the Flink engine does not support RECORD_INDEX. We are using the Flink engine for writing and the Spark engine for querying, requiring extremely high point differential efficiency. Is there any other way to add RECORD_INDEX to this table, or is it possible to enhance the Flink engine to support RECORD_INDEX? If so, what are the steps to submit this feature? This is my first time contributing code to the community. Thank you.
A clear and concise description of the problem.
**To Reproduce**
Steps to reproduce the behavior:
1.
2.
3.
4.
**Expected behavior**
A clear and concise description of what you expected to happen.
**Environment Description**
* Hudi version : 0.15.1
* Spark version : 3.3.1
* Flink Version: 1.17.1
* Hive version : 3.x
* Hadoop version : 3.1.3
* Storage (HDFS/S3/GCS..) : hdfs
* Running on Docker? (yes/no) : no
**Additional context**
Add any other context about the problem here.
**Stacktrace**
```Add the stacktrace of the error.```
Contributor guide
No contributing guide indexed for this repository
Research direction
The issue identifies the Flink engine and RECORD_INDEX but does not name files, tests, or an entry point. Start by locating the Flink engine's RECORD_INDEX handling and related Spark behavior, then define how support should be verified for Hudi 0.15.1-compatible environments.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, java
- Domain
- data-engineering, stream-processing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100