[MPP/ tidb] DAG scheduling
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Why
abstract the pipeline dependency of a query to a DAG, and schedule each pipeline one by one. From this, we can get a clear and simple view to schedule pipeline and tasks to run a query in a cluser, i.e., distributed execution. It is easy to get the execution time of each pipeline for optimization. Also, it is clear to be presented for customers. Besides, it is good for debug.
What
divide a query into many pipelines according to pipeline breakers, and abstract a DAG, then schedule each pipeline one by one.
How
break the join into two sub-operator: hash build and hash probe. In such a way, it is eacy to control the execution of each pipeline.
maintain a DAG in TiDB part
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reading the proposed pipeline-breaker approach, the hash build and hash probe split, and the note about maintaining a DAG in the TiDB part. The issue does not name files, tests, or acceptance criteria; clarify the design and define what scheduling and distributed execution must demonstrate before implementation.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 15/100