[Discussion] Multi-Region Data Locality + Federated Execution
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
Hi team,
I am looking into guarantee where compute happening in region, result to FE, no raw data crosses borders and user should be able to see aggregated result.
Please let me know if we have some work/discussion already done like this :
```
Client
│
▼
FE Leader ──── Parses SQL → generates Logical Plan → Physical Query Plan
│
│ PQP = DAG of fragments, each assigned to specific BE/CN nodes
│
├─► Fragment 1 (Scan + partial agg) ──► BE/CN-A [data local]
├─► Fragment 2 (Scan + partial agg) ──► BE/CN-B [data local]
├─► Fragment 3 (Scan + partial agg) ──► BE/CN-C [data local]
│ │ │ │
│ └──────────────┴──────────────┘
│ Exchange nodes (HASH / BROADCAST / UNPARTITIONED)
│ │
└─► Final Fragment (merge agg, top-N, ORDER BY) ──► single BE or FE itself
│
Result Set ──► back to FE ──► Client
```
Contributor guide
Research direction
No files, tests, or concrete entry points are named. Start by reviewing how the FE leader, BE/CN nodes, logical plans, physical query plans, fragments, and exchange nodes currently work. Before implementation, the discussion needs a bounded proposal with affected components and acceptance criteria for regional execution and data locality.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- sql
- Domain
- backend, databases, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100