datafusion-contrib / datafusion-contrib/datafusion-distributed
Leverage memory and network cost for dynamic task count decisions
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 139
- Forks
- 67
- Avg merge
- 3d 1h
- Merged PRs (30d)
- 35
Description
Follow-up from #432.
In prepare_dynamic_plan.rs, calculate_cost produces a cost estimation with three dimensions — cpu, memory, and network — all of which are recorded as stage metrics (cpu_cost, memory_cost, network_cost). However, compute_based_task_count currently derives the task count from cost.cpu alone:
let compute_based_task_count = cost
.cpu
.get_value()
.unwrap_or(&0)
.div_ceil(nb_ctx.d_cfg.bytes_per_partition_per_second.max(1))
.div_ceil(input_stage.plan.output_partitioning().partition_count())
.clamp(1, nb_ctx.max_tasks()?);
We should explore how to also factor the memory and network cost estimates into the task count decision (e.g. a memory-pressure-driven scale-up, or accounting for network transfer cost), rather than relying on CPU cost only.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in prepare_dynamic_plan.rs by reading calculate_cost and compute_based_task_count, then trace how the cpu, memory, and network estimates become stage metrics. Determine and document how memory and network should influence task-count decisions, and consider the existing follow-up context in issue #432. Done means task-count selection no longer relies on CPU cost alone and the behavior is validated.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100