[Feature](catalog) Support Paimon maintenance procedures and actions
- Dominant language
- Java
- Stars
- 15.9k
- Forks
- 3.9k
- Avg merge
- 2d 23h
- Merged PRs (30d)
- 520
Description
## Background
Apache Paimon exposes operational table maintenance through procedures such as compaction, snapshot expiration, rollback, orphan-file cleanup, and Branch/Tag actions. Doris currently has no Catalog-routed procedure framework for invoking these Paimon operations.
This issue tracks procedure execution and operational closure around the write capabilities in #65086. Branch/Tag domain behavior and branch writes are tracked separately; Branch/Tag procedure aliases must reuse that implementation.
Upstream references:
- https://paimon.apache.org/docs/1.4/spark/procedures/
- https://paimon.apache.org/docs/1.4/maintenance/manage-snapshots/
- https://paimon.apache.org/docs/1.4/maintenance/dedicated-compaction/
## Current Status
- Doris `CALL` supports built-in functions and Doris stored procedures, but does not resolve procedures through an External Catalog.
- Paimon/Spark-style named arguments such as `table => 'db.tbl'` are not supported by the Doris `CALL` grammar.
- Paimon maintenance operations must currently be run from another engine or SDK.
- Doris write transactions can create snapshots and files, but operators cannot complete the corresponding maintenance lifecycle from Doris SQL.
## Goals
### Catalog Procedure Framework
- Support Catalog-qualified calls such as `CALL .sys.(...)`.
- Support positional and named arguments with procedure-specific type validation and defaults.
- Resolve the target Catalog, database, table, branch, tag, and snapshot without silently falling back to another object.
- Route execution through the configured Paimon Catalog and authentication context.
- Return structured result sets and stable error codes/messages.
- Define FE forwarding, timeout, cancellation, retry, and synchronous/asynchronous execution behavior.
- Add privilege checks, audit records, sensitive-property masking, and operation metrics.
The framework should be reusable by other External Catalog implementations, while this issue's functional scope is Paimon.
### Initial Paimon Procedures
P0 operational closure:
- `compact`
- `expire_snapshots`
- `remove_orphan_files`
- `rollback_to`
- `rollback_to_as_latest`
P1 compatibility and lifecycle aliases:
- `create_tag`, `delete_tag`, and `rename_tag`
- `create_branch`, `delete_branch`, and `rename_branch`
- `fast_forward` and `merge_branch`
These Branch/Tag aliases must call the implementation owned by the Branch/Tag lifecycle issue rather than introducing a second metadata path.
P2 extended operations:
- Database-level compaction.
- Consumer maintenance.
- Partition maintenance such as mark-done or expiration actions supported by the selected Paimon version.
## Safety and Consistency Requirements
- Pin the table/ref/snapshot generation used for validation and execution.
- Detect concurrent commits and incompatible branch ancestry before destructive operations.
- Clearly distinguish `rollback_to`, which can remove later snapshots/tags, from `rollback_to_as_latest`, which publishes a new latest state.
- Expose affected snapshots/files and whether an operation made changes.
- Prevent orphan cleanup from deleting files referenced by active snapshots, tags, branches, or in-flight Doris transactions.
- Refresh Doris metadata caches after every metadata-changing procedure.
- Define bounded resource use for large compaction and cleanup operations.
## Acceptance Criteria
- Catalog-qualified procedures work against filesystem, HMS, and REST Paimon catalogs.
- Positional and named argument forms are validated consistently.
- Authorization and audit behavior are covered by regression tests.
- Concurrent write, retry, timeout, cancellation, and partial-failure cases do not produce ambiguous silent success.
- Procedure results are deterministic enough for automation and include the affected object/state.
- Cross-engine validation confirms that Doris maintenance results remain readable by Spark and Flink.
## Non-goals
- Paimon Branch DML implementation; it is tracked separately.
- General-purpose user-defined stored procedures.
- Long-running job scheduling beyond the execution model required by the initial Paimon operations.
## Related
- Paimon write tracking: #65086
- Paimon Branch/Tag lifecycle and branch writes: #67336
Contributor guide
Research direction
Start at Doris's CALL entry point and inspect how built-in functions and stored procedures are resolved, then trace the missing External Catalog path. Review the boundaries with Paimon write issue #65086 and Branch/Tag issue #67336. Done requires a defined, tested initial procedure scope with catalog resolution, validation, authorization, deterministic results, and safe failure behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, sql
- Domain
- backend-api-design, databases, observability, security
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100