apache / apache/kyuubi

[FEATURE] [SPARK] add async fetch hdfs result

Open
#7,235 4 comments 1 reaction 0 assignees View on GitHub
kind:feature priority:major
Dominant language
Scala
Stars
2.4k
Forks
1k
PR merge metrics
No merged PRs in 30d

Description

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

### Search before asking

- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.

### Describe the feature

During the Spark SQL execution phase, a thread pool is used when retrieving iterators to improve execution speed.

### Motivation

Improve the speed of Spark SQL when retrieving results from HDFS

### Describe the solution

Add a thread pool

### Additional context

_No response_

### Are you willing to submit PR?

- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.

Contributor guide

Open the contributing guide

Research direction

No file or test is named. Start by locating the Spark SQL execution path that retrieves iterators and results from HDFS, then inspect how result fetching currently runs. Done means the requested asynchronous retrieval using a thread pool is implemented and its effect on Spark SQL result-fetch speed can be verified.

Written by the indexing model from the issue text.

Assessment

Tech stack
hadoop, scala, spark
Domain
data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.