dagucloud / dagucloud/dagu

feat: resource-aware worker scheduling and admission control

Open
#2,648 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
337
Avg merge
19h 26m
Merged PRs (30d)
133

Description

## Is your feature request related to a problem?

Dagu DAGs can declare CPU and memory limits, and workers can be selected by labels and constrained by a maximum number of runs. The current resource guard is useful for limiting a process after it starts, but it does not provide scheduler-side admission control or capacity-aware placement.

A worker may therefore accept several individually valid data jobs whose combined memory, CPU, or temporary-disk demand exceeds the host. For data workloads, this often causes operating-system OOM kills, disk exhaustion, severe contention, or repeated retries. A simple run-count limit is not enough because one run may need 500 MB while another needs 30 GB.

The problem also affects mixed worker fleets: a job may require a high-memory host, local NVMe scratch space, a GPU, access to a mounted dataset, or a scarce custom resource. Labels describe compatibility but not currently available capacity.

## Describe the solution you'd like

Add resource requests, worker capacity advertisement, scheduler reservations, and admission control for local and distributed execution.

Illustrative syntax:

```yaml
resources:
requests:
cpu: "4"
memory: 16Gi
ephemeral_storage: 100Gi
custom:
ducklake_writer: 1
limits:
cpu: "8"
memory: 24Gi

steps:
- id: parse_accounts
run: python parse_accounts.py
resources:
requests:
cpu: "8"
memory: 32Gi
ephemeral_storage: 200Gi
```

The exact schema is illustrative. The desired behavior is:

1. **Worker capacity advertisement**
- Workers report total and allocatable CPU, memory, and temporary disk.
- Optional GPU and administrator-defined scalar resources.
- Capacity may be configured explicitly or detected from the host/container limits.
- A worker can be cordoned/drained without abruptly losing active work.

2. **Resource requests and limits**
- Separate scheduling requests from runtime limits.
- DAG-level defaults with optional step-level overrides.
- Required versus preferred resources/placement constraints.
- Clear units and validation for CPU, bytes, and custom resources.

3. **Atomic reservation before dispatch**
- The scheduler/coordinator reserves capacity before a worker starts the command.
- Concurrent schedulers/coordinators cannot over-reserve the same worker.
- Reservations are released on every terminal state and recovered after worker/coordinator failure.
- A stale worker cannot retain capacity indefinitely.

4. **Admission behavior**
- A run waits in a visible `waiting_for_resources`/queued state when no compatible capacity is available.
- The UI/API explains which request cannot currently be satisfied.
- Configurable behavior for impossible requests, including fail-fast after a timeout.
- Local execution should use the same model against the local host where practical.

5. **Placement and packing policy**
- A documented default such as best-fit or least-loaded placement.
- Optional anti-affinity/affinity for runs that should spread or co-locate.
- Prevent a long queue of small jobs from starving a large job indefinitely.
- Respect queue concurrency, worker selectors, runtime profiles, and child-DAG execution.

6. **Priority and fairness**
- Queue or run priority with a bounded, documented range.
- Fairness across queues/workspaces so one backfill cannot starve scheduled production jobs.
- Optional deadlines or maximum queue wait.
- Preemption is not required for an MVP; admission and fair ordering would already solve most cases.

7. **Runtime enforcement integration**
- Continue using OS/container controls for limits where supported.
- Report when a declared limit cannot be enforced instead of silently treating it as guaranteed.
- Surface OOM/resource-limit termination as a distinct failure reason.

8. **Observability**
- Worker capacity, reserved/used resources, pending demand, and placement decisions in UI/API.
- Prometheus/OpenTelemetry metrics for utilization, reservation wait, unschedulable runs, and resource-limit failures.
- Historical resource usage may be optional, but action-provided usage metadata should be attachable to a run.

## Describe alternatives you've considered

- Set a conservative `max-runs` value on every worker.
- Create separate workers/queues for small, medium, and large jobs.
- Use labels for high-memory workers and manually avoid overlap.
- Run every heavy task as a Kubernetes Job and rely on Kubernetes scheduling.
- Implement a custom external worker dispatcher.

These approaches either waste capacity, require manual tuning, or introduce infrastructure that conflicts with Dagu's lightweight deployment model. Kubernetes remains a good executor, but resource-aware scheduling should also work for a small fleet of ordinary Linux workers.

## Use case

A single machine may run frequent lightweight source checks, medium-sized transformations, and occasional 30-GB archive parsing jobs. The archive parser should wait until enough memory and scratch disk are free rather than starting beside another parser and causing both to fail. Later, the same DAG should be able to target a temporary high-memory worker without changing orchestration logic.

## Suggested acceptance criteria

- Workers expose configured/detected allocatable CPU and memory through the API.
- A DAG can declare resource requests separately from limits.
- The scheduler never dispatches work whose reservation would exceed worker allocatable capacity.
- Reservations are recovered correctly after worker/coordinator failure.
- Pending runs expose a machine-readable and human-readable unschedulable reason.
- Queue priority/fairness and worker selectors remain compatible with resource placement.
- Existing workflows without requests preserve current behavior.

Contributor guide

Open the contributing guide

Research direction

No repository files, tests, or entry points are named. Start by locating the scheduler/coordinator, worker-capacity, and API/UI paths, then split the broad proposal into a scoped MVP. Done should be measured against the listed acceptance criteria, especially capacity advertisement, atomic reservations, recovery, and explainable unschedulable states.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, prometheus, yaml
Domain
backend-api-design, distributed-systems, observability-sre
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.