apache / apache/gluten

[VL] Use multiple threads in the same executor

Open
#7,810 9 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Velox is initially designed to use multiple threads in each context. To fit spark's working model, we limit 1 thread per context. It's needed by fallback.

Since Velox backend can fully offload lots of queries, it will be interesting to investigate multiple threads per context. We can set task.cores to more than 1 then set to Velox's backend threads. Then use larger partition and see how the performance looks like.

@zhztheplayer Is there any specific consideration on memory management side?

Contributor guide

Open the contributing guide

Research direction

Start by tracing how Spark task.cores and the one-thread-per-context limit are applied, then examine how Velox backend thread counts and partition sizing are configured. Investigate memory-management considerations for multiple threads per context and compare performance with larger partitions; done means a validated approach or measured conclusion.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
data-engineering, distributed-systems, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.