apache / apache/datafusion-ballista

Ballista context should get file metadata from scheduler, not from local disk

Open
#22 3 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Rust
Stars
2.1k
Forks
320
Avg merge
1d 11h
Merged PRs (30d)
86

Description

**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
I have a Ballista cluster running, and each scheduler and executor has access to TPC-H data locally.
I am running the benchmark client on my desktop, and I do not have access to the data locally.
Query planning fails with "file not found" because `BallistaContext::read_parquet` is looking for the file on the local file system when it should be getting the file metadata from a scheduler in the cluster.

**Describe the solution you'd like**
The context should send a gRPC request to the scheduler to get the necessary metadata.

**Describe alternatives you've considered**
None

**Additional context**
None

Contributor guide

Open the contributing guide

Research direction

Start at BallistaContext::read_parquet and trace the current local file metadata lookup. Read the scheduler request path and identify the gRPC entry point needed for remote metadata. Done means a benchmark client without local TPC-H files can plan the query using metadata from the scheduler.

Written by the indexing model from the issue text.

Assessment

Tech stack
grpc, rust
Domain
distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.