dagucloud / dagucloud/dagu

Sub-DAG output variable propagation for parallel execution

Open
#1,808 0 comments 0 reactions 0 assignees View on GitHub
wishlist
Dominant language
Go
Stars
4k
Forks
332
Avg merge
19h 26m
Merged PRs (30d)
133

Description

## Problem

When a child sub-DAG sets `output:` on its steps, the parent's parallel results JSON shows `"outputs": {}` for every child. There is no way for a parent DAG to read collected outputs from parallel sub-DAG invocations.

## Expected Behavior

Child sub-DAG `output:` variables should be propagated to the parent and accessible from the parallel step's results. The parent should be able to reference aggregated outputs from all parallel children.

## Example

```yaml
# Parent
- id: fan_out
call: child-dag
parallel: ${ITEMS}
# After completion, ${fan_out.outputs} should be a JSON array
# of each child's output variables

- id: collect
depends: [fan_out]
command: echo "${fan_out.outputs}"
```

## Motivation

Fan-out/fan-in is a core DAG pattern. Without output propagation, the only way to pass data from parallel children back to the parent is through side-effect files (temp files on disk), which is fragile and hard to coordinate.

## Testing
- PR must include integration test under intg package

Contributor guide

Open the contributing guide

Research direction

Start by tracing the parallel sub-DAG execution and result aggregation described in the issue, then inspect the intg package for existing integration-test patterns. Add coverage for a child step with output variables invoked through a parent parallel step. Done means the parent’s parallel results expose an aggregated JSON array that the following step can reference.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, devops
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
55/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.