[VL] Slow scan with Spark runtime filter when split-preloading is turned on
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
In some cases when runtime filter is added, `spark.gluten.sql.columnar.backend.velox.IOThreads=0` (which turns Velox's preload / prefetch off) could gain better performance than the default settings (`spark.gluten.sql.columnar.backend.velox.IOThreads=`).
Caused by https://github.com/facebookincubator/velox/issues/10173
To temporarily bypass the issue if found, just set `spark.gluten.sql.columnar.backend.velox.IOThreads=0`.
Contributor guide
Research direction
Start by reproducing the slow scan with a Spark runtime filter while split-preloading is enabled, comparing the default IOThreads setting with IOThreads=0. Read the linked Velox issue 10173 for the reported cause; done means the performance problem is addressed or its affected conditions are clearly established.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, performance
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100