Address DynamicFilter pushdowns for path that requires merging with log
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
[https://github.com/onehouseinc/trino/pull/12#discussion_r2090063882]
As of now, DynamicFilter and all predicate pushdowns for queries involving parquet + log files merging are avoided.
Spark has some mechanisms to add some optimizations for such cases, we will need to translate the same optimizations over to Trino.
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-9425
- Type: Task
- Epic: https://issues.apache.org/jira/browse/HUDI-2687
---
## Comments
21/May/25 04:39;vhs;If MOR table has a filegroup that has no log files, we can do perform a predicate pushdown too.;;;
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked pull request discussion and the HUDI-9425 JIRA task to understand the current avoidance of DynamicFilter and predicate pushdowns for Parquet and log-file merge paths. Compare the relevant Spark optimizations and verify that Trino supports pushdown where applicable, including MOR filegroups without log files.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 42/100