apache / apache/seatunnel

[Umbrella] Lack of standardized rules for splitting long-running connector E2E tests in CI

Open
#10,995 3 comments 3 reactions 0 assignees View on GitHub
discussion
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

### Background

Currently, SeaTunnel CI suffers from long execution time and high timeout risk. A single `all-connectors-it` module can take up to **4.5 hours** to complete. If any submodule times out due to network flakiness or resource contention, the whole job gets re-run for another 4.5 hours, seriously delaying PR merge.

Some connector E2E tests are already very time-consuming:

| Module | Observed duration |
|--------|-------------------|
| E2Elasticsearch | ~62 min |
| Hbase | ~44 min |
| Clickhouse | ~42 min |
| Mongodb | ~35 min |
| CDC MySQL | ~32 min |
| Http | ~30 min |

### Problem identified from previous attempt (PR #9976)

In PR #9976, which attempted to split long-running modules into standalone jobs, we found a key issue:

**No standardized splitting rules** – There is no documented guideline on:
- **Trigger conditions** – When should a module be considered "too long" and be split out?
- **Splitting method** – How exactly should the split be done (e.g., modifying `backend.yml` and `update_modules_check.py`)?
- **Responsibility** – Who should perform the splitting? Is it suitable for new contributors, or should it be done by experienced ones?

Without such standards, different contributors may handle similar situations inconsistently, increasing review complexity and risk of configuration errors.

### Proposal

We propose to establish a **written standard for splitting long-running connector E2E tests** in CI. The standard should cover:

1. **Trigger condition** – e.g., any connector E2E module that consistently runs > 60 minutes, or any aggregated job that exceeds 2.5 hours, should be considered for splitting.
2. **Splitting method** – a step-by-step template or checklist (e.g., copy an existing standalone job like `elasticsearch-connector-it`, adjust module names, and update both `backend.yml` and `update_modules_check.py` accordingly).
3. **Responsibility** – clarify whether this should be done by core contributors (due to CI complexity) or can be done by anyone with proper guidance, and who should review such changes.

### Next steps / discussion points

- What should the exact thresholds be (e.g., 60 min per module, 2.5h per parent job)?
- Should we write a short guide / PR template checklist?
- Who will lead the effort to draft and document the standard?

Contributor guide

No contributing guide indexed for this repository

Research direction

Read the existing standalone job such as elasticsearch-connector-it, then compare the CI configuration in backend.yml with the module handling in update_modules_check.py and the approach from PR #9976. Done means a written standard covering split thresholds, the step-by-step update process, contributor responsibility, and review ownership.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ci-cd, documentation
Issue type
Documentation
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.