dagucloud / dagucloud/dagu

Keep container/-step alive across dags (when referencing as sub-dag)

Open
#1,364 1 comment 0 reactions 1 assignee Claimed by @yohamta0 View on GitHub
Dominant language
Go
Stars
4k
Forks
332
Avg merge
19h 26m
Merged PRs (30d)
133

Description

Hello, I just started to play around with dagu, and I am really liking it so far, but I seemingly can't get the following scenario to work.

I would like to create a container or step that specifies spins up a postgres-db, and exposes the ports in the host network. Ideally, this container/step can be included as a sub-dag in other dags.

I've started from the postgres-example given in #1244, which works perfectly.

That example I've changed to the following dag named `postgres`:
```yaml
container:
image: postgres:17
env:
- POSTGRES_PASSWORD=secret
- POSTGRES_DB=appdb
keepContainer: true
network: host
ports:
- 5432:5432
startup: entrypoint
waitFor: running

steps:
- name: check-ready
command: pg_isready -U postgres -d appdb
retryPolicy:
limit: 10
intervalSec: 1
```

From this [example](https://docs.dagu.io/writing-workflows/examples#keep-container-running) and the [note on the container-configuration](https://docs.dagu.io/reference/yaml#container-configuration) I assume that the container is kept alive atleast until the sub-dag is fully executed. The initial test with the example given in #1244 seems to confirm this.

However, if I call the postgres-dag in another dag - let's call it `postgres_create_table` - I seemingly can't connect to the postgres-db.
The dag-spec for 'postgres_create_table` is as follows:
```yaml
params:
- CREATE_TABLE: "'CREATE TABLE IF NOT EXISTS demo(id INT PRIMARY KEY, name TEXT);'"

env:
- PGPASSWORD: secret

steps:
- call: postgres.yaml
depends: []

- name: create-db
executor:
type: docker
config:
image: alpine
autoRemove: true
command: sh -c "apk add postgresql17-client --allow-untrusted && psql -U postgres -h localhost -d appdb -c ${CREATE_TABLE}"
```

The error message is as follows:
```
* Setting postgresql17 as the default version
psql: error: connection to server at "localhost" (::1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
connection to server at "localhost" (127.0.0.1), port 5432 failed: Connection refused
Is the server running on that host and accepting TCP/IP connections?
```

If I run `netstat` instead of the `create-db`-step, it yields an empty list.

I also tried changing the `startup`-property of `posgtres`-dag to `keepalive` and manually trigger the entrypoint-script and the startup of the postgres-image. This fails either due to inappropriate rights (sudo-user starting postgres) or the `postgresql.conf` not being available after explicitly changing the user via `su -c postgres postgres` with the following error message:
```
postgres: could not access the server configuration file "/var/lib/postgresql/data/postgresql.conf": No such file or directory
```

Am I making an obvious mistake, or is it simply not possible as of now to keep a container/step alive across dags? Thanks for any response in advance!

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.