apache / apache/seatunnel

[Discussion][Core] Define a unified validate, explain, and plan contract before job submission

Open
#11,405 1 comment 0 reactions 0 assignees View on GitHub
core design discussion feature
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

## Background
SeaTunnel still lacks a unified contract for what should happen **before** a job is submitted.

Today, several related needs appear separately:
- static config validation
- semantic connector validation
- dry-run expectations
- explainability for generated job structure
- reusable plan output for later platform or UI consumption

These are related, but the product contract is still fragmented.

## Problem to solve
We should define what users and platforms can expect from a pre-run workflow before job submission.

That workflow should answer at least three questions:
1. Is this job valid?
2. What does this job mean structurally?
3. What plan will SeaTunnel execute?

## Proposal
Define a unified pre-run contract with three complementary concepts:
- validate
- explain
- plan

Suggested first-delivery scope:
- define a minimal validate contract
- define minimal explain output for pipeline structure and intent
- define minimal plan output that can later be reused by CLI, REST, and platform integrations

## Why this should be discussion-first
This topic affects:
- CLI behavior
- REST/API behavior
- core planning and validation flow
- future platform and automation integration

There is already related work around dry-run and structured validation, but this discussion is intentionally broader: the goal is to align those efforts under one stable pre-run contract instead of leaving them as separate point features.

## Non-goals for v1
- do not build a complete operations platform
- do not replace every runtime check
- do not require full connectivity sampling in the first delivery

## Suggested validation
- validate common invalid config scenarios
- validate semantic mismatch scenarios
- validate that explain output helps a user understand the pipeline structure
- validate that plan output is stable enough for future reuse

## Acceptance criteria
- validate, explain, and plan each have clear scope boundaries
- the first delivery stays intentionally small
- the design can absorb existing dry-run and validation efforts instead of duplicating them
- the community agrees on what belongs in the pre-run contract and what remains runtime-only

## Related issues
- #10247
- #10681

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reviewing related issues #10247 and #10681, then compare the existing dry-run and structured validation efforts mentioned in the discussion. Done means the community agrees on clear boundaries for validate, explain, and plan, a deliberately small first delivery, and which checks remain runtime-only.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
api, backend-api-design, cli, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.