apache / apache/gluten

[VL] Slow scan with Spark runtime filter when split-preloading is turned on

Open
#6,074 0 comments 2 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

In some cases when runtime filter is added, `spark.gluten.sql.columnar.backend.velox.IOThreads=0` (which turns Velox's preload / prefetch off) could gain better performance than the default settings (`spark.gluten.sql.columnar.backend.velox.IOThreads=`).

Caused by https://github.com/facebookincubator/velox/issues/10173

To temporarily bypass the issue if found, just set `spark.gluten.sql.columnar.backend.velox.IOThreads=0`.

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the slow scan with a Spark runtime filter while split-preloading is enabled, comparing the default IOThreads setting with IOThreads=0. Read the linked Velox issue 10173 for the reported cause; done means the performance problem is addressed or its affected conditions are clearly established.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, performance
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.