Support spill to disk for memory intensive operator
Open
Nobody has claimed this yet.
type/enhancement
- Dominant language
- C++
- Stars
- 1k
- Forks
- 423
- Avg merge
- 1d 15h
- Merged PRs (30d)
- 24
Description
Enhancement
Support spill to disk for memory intensive operator like join/aggregation/sort/topN/window.
Pre-work
- Make
max_block_sizework for all InputStreams https://github.com/pingcap/tiflash/issues/3436
Sub-tasks
- Framework
- Memory tracker for MPP task/query/query group
- Spill framework
- Spill algorithm for memory intensive operators
- Sort/TopN
- (Hash)Aggregation
- (Hash)Join
- Window
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Review the prerequisite issue #3436 and the listed framework and operator subtasks first. Completion is not defined beyond supporting disk spilling for join, aggregation, sort, TopN, and window operators, so a maintainer must identify the first slice and relevant files or tests before work begins.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- cpp
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100