apache / apache/datafusion-ballista
Introduce a source data cache layer
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 22h
- Merged PRs (30d)
- 66
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
In a cloud native architecture with completely stateless executors, each executor needs to fetch the source data from the remote storage. If the amount of source data is very large, it will easily meet the network throughput bottleneck and it will take too much time for this step of fetching source data. For example, for a compute layer with 20 nodes with 10Gb node bandwidth, it will take at least 4s to fetch 100GB source data. While it often takes less than 1s to finish other steps of processing the 100GB data. Therefore, it’s better to introduce a cache layer into the cloud native architecture to make the executors be of weak state for caching hot data on local disk, like [snowflake](https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers/p215-dageville-snowflake.pdf) does.
**Describe the solution you'd like**
https://docs.google.com/document/d/1iMFv3S-TuiwBoTzp4KX0Ltrrenm86ULr0q_PwIKdW6g/edit?usp=sharing
To achieve this goal, we need to finish the following tasks:
- [x] (Executor) Introduce a 2-tiered cache manager for caching data on remote storage based on local memory and disk
- [x] #652
- [x] #825
- [x] #827
- [x] (Scheduler) #830
- [ ] (Scheduler) #833
- [x] (Scheduler) #648, #649
**Describe alternatives you've considered**
**Additional context**
Contributor guide
Research direction
Start with the linked design document and the unfinished Scheduler task (#833), then review the completed executor and scheduler items listed in the issue. The issue names no files or tests; done means completing the remaining scheduler portion of the source-data cache plan tracked by #833.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100