datafusion-contrib / datafusion-contrib/datafusion-distributed

Leverage memory and network cost for dynamic task count decisions

Open
#520 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
139
Forks
67
Avg merge
3d 1h
Merged PRs (30d)
35

Description

Follow-up from #432.

In prepare_dynamic_plan.rs, calculate_cost produces a cost estimation with three dimensions — cpu, memory, and network — all of which are recorded as stage metrics (cpu_cost, memory_cost, network_cost). However, compute_based_task_count currently derives the task count from cost.cpu alone:

let compute_based_task_count = cost
    .cpu
    .get_value()
    .unwrap_or(&0)
    .div_ceil(nb_ctx.d_cfg.bytes_per_partition_per_second.max(1))
    .div_ceil(input_stage.plan.output_partitioning().partition_count())
    .clamp(1, nb_ctx.max_tasks()?);

We should explore how to also factor the memory and network cost estimates into the task count decision (e.g. a memory-pressure-driven scale-up, or accounting for network transfer cost), rather than relying on CPU cost only.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in prepare_dynamic_plan.rs by reading calculate_cost and compute_based_task_count, then trace how the cpu, memory, and network estimates become stage metrics. Determine and document how memory and network should influence task-count decisions, and consider the existing follow-up context in issue #432. Done means task-count selection no longer relies on CPU cost alone and the behavior is validated.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
distributed-systems
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.