Worker Sync Mode for Remote Storage
- Dominant language
- Go
- Stars
- 4k
- Forks
- 332
- Avg merge
- 19h 26m
- Merged PRs (30d)
- 133
Description
Currently a distributed DAG run only succeeds when workers, coordinators, schedulers, and the UI server share the same filesystem tree.
A worker polled by `internal/worker` ultimately shells out to `dagu start`, whose `internal/agent` writes attempt metadata through the file-backed stores in `internal/persistence/filedagrun`, persists heartbeats in `internal/persistence/fileproc`, streams logs under the configured log directory. The coordinator (`internal/coordinator`) merely matches pollers with tasks; it never persists run state. As a result, running `dagu worker` on a host without the shared volume breaks status history, log access, stop/retry controls.
**Requirements**
- Allow `dagu worker --sync` to run on a host without access to the server's persistent filesystem while keeping identical behaviour for schedulers, coordinators, UI, and CLI consumers.
- Ensure all state mutations produced by `internal/agent` (attempt creation, status writes, DAG definition snapshots, retention cleanup) land in the server-side `filedagrun.Store` within hundreds milliseconds.
- Preserve live log access and historical artifacts by continuously synchronizing scheduler and node logs on the server.
- Maintain process visibility and control operations (stop, retry, cancel) from the server even though the worker runs remotely.
- Make the solution resilient to transient network failures and coordinator restarts without losing run data.
Contributor guide
Assessment
This issue has not been assessed yet.