dagucloud / dagucloud/dagu

feat: official DuckLake action with transactional publication metadata

Open
#2,652 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Go
Stars
4k
Forks
337
Avg merge
19h 26m
Merged PRs (30d)
133

Description

## Is your feature request related to a problem?

Dagu already has an official `duckdb@v1` action, so a workflow can execute the SQL needed to attach and use DuckLake. A production DuckLake write still requires every DAG to recreate orchestration-sensitive behavior around that SQL:

- install/load and pin a compatible DuckDB/DuckLake extension version;
- construct an `ATTACH` configuration for a catalog database and local/object-store data path;
- inject catalog and object-store credentials without leaking them into logs;
- define transaction and rollback boundaries;
- identify the DuckLake snapshot committed by this step;
- distinguish transient commit conflicts from logical conflicts;
- attach run/step/source metadata to the commit;
- expose the committed snapshot to downstream steps, artifacts, quality gates, and lineage;
- apply consistent memory/temp-directory/runtime settings on different workers.

A generic SQL action is intentionally flexible, but it cannot provide a stable DuckLake-specific result contract or safe publication conventions. Reimplementing those conventions in shell/Python wrappers makes retries and observability inconsistent.

## Describe the solution you'd like

Add an official versioned `ducklake@v1` action, potentially implemented as a specialization/wrapper around the existing DuckDB action.

Illustrative usage:

```yaml
steps:
- id: publish_companies
uses: ducklake@v1
with:
catalog:
connection: ${DUCKLAKE_CATALOG_DSN}
alias: lake
data_path: s3://company-data/lake/
sql_file: sql/publish_companies.sql
transaction: true
commit:
author: "dagu:${DAG_NAME}"
message: "${DAG_RUN_ID}: publish canonical companies"
extra:
source_release: ${SOURCE_RELEASE}
conflict_retry:
limit: 5
backoff: exponential
settings:
memory_limit: 24GB
temp_directory: /scratch/duckdb
```

The exact input schema is illustrative. The requested behavior is:

1. **DuckLake connection and attachment**
- Support local catalog files and SQL catalog backends supported by DuckLake.
- Support local and S3-compatible data paths.
- A stable alias/database selection model for SQL files.
- Validate required catalog/data-path combinations before executing the query.
- Allow read-only attachment for consumers and write attachment for producers.

2. **Secure credential handling**
- Accept Dagu secret references for catalog and object-store credentials.
- Configure DuckDB secrets/settings without printing resolved values in logs, errors, generated SQL, or outputs.
- Avoid placing credentials directly in process arguments where practical.
- Document which credentials are forwarded to distributed workers.

3. **Version and dependency control**
- Pin or declare supported DuckDB and DuckLake extension versions.
- Use Dagu action dependency snapshots so local and remote workers run compatible binaries/extensions.
- Cache extension downloads safely while allowing an offline/preinstalled mode.
- Fail clearly when the requested version is unavailable or incompatible.

4. **Transaction/publication semantics**
- Optionally wrap the supplied SQL in one explicit transaction.
- On failure or cancellation, roll back and do not report a successful publication.
- Define behavior for SQL that manages its own transaction.
- Capture the snapshot before execution and the snapshot committed by this connection/transaction.
- Treat the committed DuckLake snapshot as the publication version returned by the action.

5. **Commit attribution and provenance**
- Set an optional DuckLake commit author, message, and bounded extra metadata from Dagu context.
- Include DAG name, run ID, step ID, attempt, trigger, partition/item key, source version, and code/spec revision where configured.
- Do not silently truncate or expose sensitive parameters; validate metadata size and redaction.

6. **Typed outputs**
- At minimum: catalog/alias, snapshot before, committed snapshot ID, commit timestamp where available, success/read-only status, query duration, and affected result metadata.
- Optional changed schema/table summary when DuckLake exposes it reliably.
- Query result support consistent with `duckdb@v1`, with size limits and artifact output for large results.
- Outputs should be usable by downstream steps and a future asset/materialization model such as #2644.

7. **Conflict-aware retries**
- Recognize DuckLake commit conflicts separately from arbitrary SQL/application failures.
- Retry only conflicts documented as safe/transient, using bounded configurable backoff and jitter.
- Do not blindly retry logical conflicts, destructive DDL conflicts, or non-idempotent external side effects.
- Return structured conflict information and the competing/latest snapshot where available.
- Allow integration with a future named lock primitive such as #2647 for workflows that deliberately require a single writer.

8. **Resource and runtime settings**
- Expose common DuckDB controls such as memory limit, thread count, temp directory, and extension directory without requiring setup SQL.
- Validate scratch-path availability and make requirements visible to worker selection/resource admission.
- Preserve a generic `settings` escape hatch with allow/deny/redaction rules.

9. **Quality and publication integration**
- Allow post-query checks to run before the action is considered published where transaction semantics permit it.
- Alternatively, return the snapshot as a candidate and let a Dagu quality gate determine whether it becomes the latest advertised asset version.
- Emit row counts/check results/metadata without requiring log parsing.

10. **Maintenance operations**
- The MVP can focus on transactional SQL execution.
- A later operation mode may expose common DuckLake maintenance such as compaction, snapshot inspection, expiration, and orphan-file cleanup with safe defaults, dry-run support, and structured results.

11. **Observability**
- Log the catalog alias, operation mode, query file/hash, retry count, and committed snapshot, but not secrets or full sensitive SQL by default.
- Emit action metrics for duration, conflicts/retries, rows/bytes where known, and snapshot ID as run metadata rather than a high-cardinality metric label.
- Link directly from the Dagu step to returned snapshot metadata.

## Describe alternatives you've considered

- Use `duckdb@v1` with manual `INSTALL/LOAD/ATTACH/BEGIN/COMMIT` SQL.
- Call the DuckDB CLI from a shell step.
- Build a project-specific Python wrapper around DuckDB/DuckLake.
- Use dbt with a DuckDB/DuckLake profile.
- Put all writes behind a separate service.

All are workable. The official action would not replace generic SQL or dbt; it would standardize the small but important boundary between Dagu run semantics and DuckLake snapshot/transaction semantics.

## Use case

A low-operations data platform uses a PostgreSQL-backed DuckLake catalog and S3-compatible Parquet storage. Source ingestions and canonical transformations run on one VPS today and may later use temporary distributed workers. Every successful publish should create one attributable DuckLake snapshot and return its ID; failed or cancelled attempts must not be advertised as current data.

The action would also benefit analytics, ML feature stores, archival pipelines, and other Dagu users adopting DuckLake as a lightweight lakehouse.

## Suggested acceptance criteria

- `ducklake@v1` can attach a DuckLake catalog/data path and execute a SQL file on local and distributed workers.
- Credentials are accepted through Dagu secrets and are redacted from logs and outputs.
- With transaction mode enabled, a failed/cancelled action rolls back and returns no successful committed snapshot.
- A successful write returns the snapshot committed by that connection plus run/step provenance metadata.
- Transient commit conflicts can be retried with bounded policy, while logical conflicts are surfaced distinctly.
- DuckDB/DuckLake dependency versions are reproducible across workers.
- The action remains compatible with generic `duckdb@v1`; users can choose either.

Contributor guide

Open the contributing guide

Research direction

Start by locating the existing duckdb@v1 action and its action or worker execution entry points; the issue names no specific files or tests. Use the acceptance criteria to scope an MVP around transactional SQL execution, then verify attachment, secret redaction, rollback, committed snapshot outputs, dependency reproducibility, and conflict handling on local and distributed workers.

Written by the indexing model from the issue text.

Assessment

Tech stack
go, sql
Domain
data-engineering, devops
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.