baidu / baidu/bigflow

API Plan optimizations is needed

Open
#4 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
C++
Stars
1.1k
Forks
159
PR merge metrics
No merged PRs in 30d

Description

We should add a optimization layer in the current API layer, it could be called API Plan layer.

At this moment, API layer will transform the user's code to a LogicalPlan directly, and some information is lost, such as LogicalPlan don't know what is `join`, the LogicalPlan only know that two nodes are cogrouped and then a Processor will process the two cogrouped result.

Eg.

```pc1.distinct().join(pc2)```

is equal to

```
pc1.cogroup(pc2) \
.apply_values(lambda p1, p2: p1.distinct().cartesian(p2)) \
.flatten_values()
```

But we can't optimize it automatically without the help of API Plan.

So, API Plan is meant to keep all the information we can get from user's code, and optimize the plan by the information.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the current API layer and how it transforms user code into LogicalPlan, then trace how Processor handles cogrouped results. Compare the join example with its cogroup expansion and define what information API Plan must retain and how optimization should be evaluated; the issue does not name files or tests.

Written by the indexing model from the issue text.

Assessment

Tech stack
cpp
Domain
api, distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.