[VL] Use multiple threads in the same executor
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Velox is initially designed to use multiple threads in each context. To fit spark's working model, we limit 1 thread per context. It's needed by fallback.
Since Velox backend can fully offload lots of queries, it will be interesting to investigate multiple threads per context. We can set task.cores to more than 1 then set to Velox's backend threads. Then use larger partition and see how the performance looks like.
@zhztheplayer Is there any specific consideration on memory management side?
Contributor guide
Research direction
Start by tracing how Spark task.cores and the one-thread-per-context limit are applied, then examine how Velox backend thread counts and partition sizing are configured. Investigate memory-management considerations for multiple threads per context and compare performance with larger partitions; done means a validated approach or measured conclusion.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, spark
- Domain
- data-engineering, distributed-systems, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100