dagucloud / dagucloud/dagu

Queues are not used when DAG is called as sub dag

Open
#2,047 9 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
332
Avg merge
19h 26m
Merged PRs (30d)
133

Description

Hi everyone ;
The following has been tested against dagu 2.6.1.

I have defined a queue in `~/.config/dagu/config.yaml` :

```yaml
queues:
enabled: true
config:
- name: "my-queue"
max_concurrency: 1
```

I have a main DAG which starts sub DAG :

```yaml
type: graph
steps:
- id: my_step
call: my_sub_dag
parallel:
items:
- item1
- item2
- item3
- item4
- item5
- item6
- item7
params:
SCOPE: "${ITEM}"
---
name: my_sub_dag
type: graph
queue: my-queue
steps:
- id: run_this
command: echo "calling this with scope ${SCOPE}" && /bin/bash -c 'sleep "$((1 + $RANDOM % 10))"'
```

Expected behavior : `my_sub_dag` should run one at a time, regarding the `max_concurrency` specified by the queue definition
Current behavior : `my_sub_dag` is all run at the same time (7 occurrences in the example above)

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the example with ~/.config/dagu/config.yaml and the shown main and sub-DAG definitions, using seven parallel calls and max_concurrency: 1. Trace how queue settings are applied when a sub-DAG is invoked and verify that concurrent sub-DAG executions share the configured queue; done means only one my_sub_dag instance runs at a time.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.