[FEATURE] [SPARK] add async fetch hdfs result
- Dominant language
- Scala
- Stars
- 2.4k
- Forks
- 1k
- PR merge metrics
- No merged PRs in 30d
Description
### Code of Conduct
- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)
### Search before asking
- [x] I have searched in the [issues](https://github.com/apache/kyuubi/issues?q=is%3Aissue) and found no similar issues.
### Describe the feature
During the Spark SQL execution phase, a thread pool is used when retrieving iterators to improve execution speed.
### Motivation
Improve the speed of Spark SQL when retrieving results from HDFS
### Describe the solution
Add a thread pool
### Additional context
_No response_
### Are you willing to submit PR?
- [x] Yes. I would be willing to submit a PR with guidance from the Kyuubi community to improve.
- [ ] No. I cannot submit a PR at this time.
Contributor guide
Research direction
No file or test is named. Start by locating the Spark SQL execution path that retrieves iterators and results from HDFS, then inspect how result fetching currently runs. Done means the requested asynchronous retrieval using a thread pool is implemented and its effect on Spark SQL result-fetch speed can be verified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- hadoop, scala, spark
- Domain
- data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 30/100