dagucloud / dagucloud/dagu

Worker Sync Mode for Remote Storage

Open
#1,260 5 comments 8 reactions 1 assignee Claimed by @yohamta0 View on GitHub
Dominant language
Go
Stars
4k
Forks
332
Avg merge
19h 26m
Merged PRs (30d)
133

Description

Currently a distributed DAG run only succeeds when workers, coordinators, schedulers, and the UI server share the same filesystem tree.
A worker polled by `internal/worker` ultimately shells out to `dagu start`, whose `internal/agent` writes attempt metadata through the file-backed stores in `internal/persistence/filedagrun`, persists heartbeats in `internal/persistence/fileproc`, streams logs under the configured log directory. The coordinator (`internal/coordinator`) merely matches pollers with tasks; it never persists run state. As a result, running `dagu worker` on a host without the shared volume breaks status history, log access, stop/retry controls.

**Requirements**
- Allow `dagu worker --sync` to run on a host without access to the server's persistent filesystem while keeping identical behaviour for schedulers, coordinators, UI, and CLI consumers.
- Ensure all state mutations produced by `internal/agent` (attempt creation, status writes, DAG definition snapshots, retention cleanup) land in the server-side `filedagrun.Store` within hundreds milliseconds.
- Preserve live log access and historical artifacts by continuously synchronizing scheduler and node logs on the server.
- Maintain process visibility and control operations (stop, retry, cancel) from the server even though the worker runs remotely.
- Make the solution resilient to transient network failures and coordinator restarts without losing run data.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.