Investigate records returned when full table scan is enabled for incremental query
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
HUDI-6649 adds assertion for SQL queries with column stat index enabled. One of the assertions in the test class `org.apache.hudi.functional.TestColumnStatsIndexWithSQL#verifyFileIndexAndSQLQueries` is failing when full table scan is enabled. If the full table scan is disabled, the incremental query provides the correct results.
The Jira aims to investigate and fix the query results.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-6657
- Type: Bug
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with org.apache.hudi.functional.TestColumnStatsIndexWithSQL#verifyFileIndexAndSQLQueries and reproduce the failing assertion with full table scan enabled. Compare the incremental query records in both scan modes and trace the column stats index or file-index path involved. Done means the query returns the correct records and the assertion passes with full table scans enabled.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark, sql
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100