API Plan optimizations is needed
- Dominant language
- C++
- Stars
- 1.1k
- Forks
- 159
- PR merge metrics
- No merged PRs in 30d
Description
We should add a optimization layer in the current API layer, it could be called API Plan layer.
At this moment, API layer will transform the user's code to a LogicalPlan directly, and some information is lost, such as LogicalPlan don't know what is `join`, the LogicalPlan only know that two nodes are cogrouped and then a Processor will process the two cogrouped result.
Eg.
```pc1.distinct().join(pc2)```
is equal to
```
pc1.cogroup(pc2) \
.apply_values(lambda p1, p2: p1.distinct().cartesian(p2)) \
.flatten_values()
```
But we can't optimize it automatically without the help of API Plan.
So, API Plan is meant to keep all the information we can get from user's code, and optimize the plan by the information.
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the current API layer and how it transforms user code into LogicalPlan, then trace how Processor handles cogrouped results. Compare the join example with its cogroup expansion and define what information API Plan must retain and how optimization should be evaluated; the issue does not name files or tests.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- api, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100