apache / apache/datafusion-ballista

Introduce a source data cache layer

Open
#645 2 comments 3 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 22h
Merged PRs (30d)
66

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**

In a cloud native architecture with completely stateless executors, each executor needs to fetch the source data from the remote storage. If the amount of source data is very large, it will easily meet the network throughput bottleneck and it will take too much time for this step of fetching source data. For example, for a compute layer with 20 nodes with 10Gb node bandwidth, it will take at least 4s to fetch 100GB source data. While it often takes less than 1s to finish other steps of processing the 100GB data. Therefore, it’s better to introduce a cache layer into the cloud native architecture to make the executors be of weak state for caching hot data on local disk, like [snowflake](https://pages.cs.wisc.edu/~remzi/Classes/739/Fall2018/Papers/p215-dageville-snowflake.pdf) does.

**Describe the solution you'd like**

https://docs.google.com/document/d/1iMFv3S-TuiwBoTzp4KX0Ltrrenm86ULr0q_PwIKdW6g/edit?usp=sharing

To achieve this goal, we need to finish the following tasks:
- [x] (Executor) Introduce a 2-tiered cache manager for caching data on remote storage based on local memory and disk
- [x] #652
- [x] #825
- [x] #827
- [x] (Scheduler) #830
- [ ] (Scheduler) #833
- [x] (Scheduler) #648, #649

**Describe alternatives you've considered**

**Additional context**

Contributor guide

Open the contributing guide

Research direction

Start with the linked design document and the unfinished Scheduler task (#833), then review the completed executor and scheduler items listed in the issue. The issue names no files or tests; done means completing the remaining scheduler portion of the source-data cache plan tracked by #833.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.