feat: first-class partitions and resumable backfills
- Dominant language
- Go
- Stars
- 4k
- Forks
- 337
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 133
Description
## Is your feature request related to a problem?
Dagu supports schedules, catch-up windows, parameters, parallel child DAGs, and retries. These are useful primitives, but they do not model a data partition as a durable unit of work.
For historical and incremental data pipelines, an operator often needs to process logical partitions such as:
- one day, month, filing year, country, or registry release;
- a static list of source identifiers;
- dynamically discovered batches or object-store prefixes;
- a multi-dimensional key such as `country=GB/year=2025`.
Today a backfill must be implemented by generating parameters or child DAG calls manually. Dagu does not provide a first-class view of which partitions are complete, missing, failed, stale, or currently in progress. Restarting a large historical run can therefore create duplicates or require custom bookkeeping.
#695 requests scheduled runs between a start and end date, and #879 proposes matrix execution. Those are related, but a data backfill also needs durable per-partition state, resumability, bounded concurrency, and operational controls.
## Describe the solution you'd like
Add first-class partition definitions and a backfill operation. An illustrative time-window definition could be:
```yaml
partitions:
type: time_window
start: 2020-01-01
cadence: daily
timezone: UTC
steps:
- id: ingest
run: python ingest.py \
--start "${DAGU_PARTITION_START}" \
--end "${DAGU_PARTITION_END}"
```
A static or multi-dimensional example could be:
```yaml
partitions:
type: static
values:
- country=BE/year=2024
- country=NL/year=2024
- country=GB/year=2024
```
The exact syntax is illustrative. The desired behavior is:
1. **Partition definitions**
- Time-window partitions with start, cadence, timezone, and optional end.
- Static keys.
- Custom/dynamic keys supplied through API, CLI, another step, or a future event trigger.
- A canonical partition key plus structured partition dimensions.
2. **Partition context**
- Expose the key and dimensions through typed parameters/environment variables.
- For time windows, expose logical start/end timestamps independently of wall-clock execution time.
- Propagate partition context to child DAGs, outputs, artifacts, state, and future asset materializations.
3. **Backfill planning and submission**
- CLI, API, and UI selection by range, explicit keys, or predicate.
- A dry plan showing the partitions that would run and those that will be skipped.
- Options such as `missing`, `failed`, `stale`, `all`, and `force`.
- Deterministic/idempotent submission so repeated requests do not accidentally enqueue the same partition twice.
4. **Durable per-partition status**
- Track queued, running, succeeded, failed, aborted, skipped, and missing status per partition.
- Link each partition attempt to its DAG run and artifacts.
- Preserve history while exposing the latest effective status.
5. **Bounded execution**
- Backfill-level maximum concurrency and rate limits.
- Respect DAG queues, `max_active_runs`, worker selectors, and distributed execution.
- Avoid expanding millions of partitions into memory at once; submit/process them in bounded batches.
6. **Operational controls**
- Pause, resume, cancel, retry failed partitions, and re-run a selected subset.
- A backfill summary with total/planned/running/succeeded/failed/skipped counts and ETA where possible.
- Resume after scheduler/coordinator restart without losing the plan or duplicating completed partitions.
7. **Scheduling integration**
- Scheduled and catch-up runs should use the same partition identity as manual backfills.
- Overlap policies should operate on logical partition keys where appropriate.
- A late scheduled run and a manual backfill for the same partition should be deduplicated or explicitly forced.
## Describe alternatives you've considered
- Generate one child DAG call per date or source key with `parallel.items`.
- Use shell loops or an external script to call the Dagu API repeatedly.
- Store completed keys manually through Dagu state actions.
- Encode the logical date in a custom DAG-run ID.
- Use schedule catch-up for recent periods and a separate ad hoc DAG for older history.
These approaches can work for small ranges, but every pipeline must reimplement planning, deduplication, progress tracking, partial retry, and recovery.
## Use case
A company-data platform may need to process twenty years of annual filings for several countries, then ingest one new daily release at a time. A historical run can contain tens of thousands of logical partitions and take days. If five partitions fail, the operator should be able to retry only those five and retain a clear record of the rest.
The same feature is useful for warehouse loads, log processing, ML training windows, media transcoding, backups, and any workflow where the logical work unit differs from the physical run time.
## Suggested acceptance criteria
- A DAG can define time-window and static partitions.
- A backfill can be created through API and CLI for a selected partition range/list.
- Dagu records and exposes status per partition and links it to DAG runs.
- Re-submitting the same backfill request does not create duplicate active work unless forced.
- Failed or missing partitions can be retried without re-running successful ones.
- Backfills use bounded concurrency and survive a Dagu restart.
- Existing schedules and non-partitioned DAGs remain backward compatible.
Contributor guide
Research direction
No implementation files or tests are named. Start by tracing Dagu's existing scheduling, catch-up, child-DAG, state, API, and CLI paths, then define how partition identity and durable status fit them; done means the listed acceptance criteria are met without breaking existing schedules or non-partitioned DAGs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend-api-design, cli, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100