apache / apache/datafusion-ballista
Ballista context should get file metadata from scheduler, not from local disk
- Dominant language
- Rust
- Stars
- 2.1k
- Forks
- 320
- Avg merge
- 1d 11h
- Merged PRs (30d)
- 86
Description
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I have a Ballista cluster running, and each scheduler and executor has access to TPC-H data locally.
I am running the benchmark client on my desktop, and I do not have access to the data locally.
Query planning fails with "file not found" because `BallistaContext::read_parquet` is looking for the file on the local file system when it should be getting the file metadata from a scheduler in the cluster.
**Describe the solution you'd like**
The context should send a gRPC request to the scheduler to get the necessary metadata.
**Describe alternatives you've considered**
None
**Additional context**
None
Contributor guide
Research direction
Start at BallistaContext::read_parquet and trace the current local file metadata lookup. Read the scheduler request path and identify the gRPC entry point needed for remote metadata. Done means a benchmark client without local TPC-H files can plan the query using metadata from the scheduler.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- grpc, rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100