apache / apache/seatunnel

[Improve][Zeta] Investigate and improve job lifecycle growth benchmark stability

Open
#12,063 27 comments 0 reactions 0 assignees View on GitHub
help wanted performance
Dominant language
Java
Stars
9.7k
Forks
2.4k
Avg merge
3d 9h
Merged PRs (30d)
204

Description

## Description

This is a focused investigation and improvement task for the latency variance observed in the IMap job-lifecycle growth benchmarks.

It is not a performance regression report.

The benchmark results show noticeable sample-to-sample variance in both running-job growth and completed-job history growth:

| Benchmark | Parameters | Score | Error | CV |
| --- | --- | ---: | ---: | ---: |
| `runningJobGrowth` | `initialStoredJobCount=0` | 410.975 us/op | 15.74% | 14.72% |
| `runningJobGrowth` | `initialStoredJobCount=1000` | 342.508 us/op | 10.58% | 9.90% |
| `completedJobHistoryGrowth` | `initialStoredJobCount=0` | 2,219.33 us/op | 10.63% | 9.94% |
| `completedJobHistoryGrowth` | `initialStoredJobCount=1000` | 2,015.11 us/op | 10.03% | 9.38% |

Benchmark run:

https://github.com/apache/seatunnel/actions/runs/33750118527

Other benchmarks executed in the same environment produced substantially lower variance. These two methods also share the same job-lifecycle growth workload, so they should be investigated together.

The goals of this issue are to:

1. Identify whether the variance comes from the benchmark fixture, Hazelcast IMap operations, serialization/allocation, WAL/FileMapStore persistence, lifecycle operations, or the execution environment.
2. Improve the benchmark fixture or methodology if it causes the instability.
3. Optimize the production job-lifecycle storage path if profiling confirms an implementation bottleneck.
4. Preserve running-job and completed-job state correctness and durability.

## Relevant benchmarks

```text
IMapJobStorageBenchmark.runningJobGrowth
IMapJobStorageBenchmark.completedJobHistoryGrowth
```

## Profiling tools

For an overview of the Zeta benchmark suite and usage, see the [SeaTunnel Zeta Benchmark Guide](https://seatunnel.apache.org/docs/engines/zeta/benchmark).

SeaTunnel provides the `Benchmarks Diagnostics` workflow for running one exact benchmark method with CPU, wall-clock, lock, GC, or JFR profiling:

https://github.com/apache/seatunnel/actions/workflows/benchmarks_diagnostics.yml

The benchmark can also be run directly from the GitHub Actions page by selecting **Benchmarks Diagnostics**, clicking **Run workflow**, and providing the target branch, tag, commit SHA, or trusted PR number together with the exact benchmark method.

## Local profiling

Build the benchmark module:

```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
```

Run a profiler locally:

```bash
bash tools/benchmarks/profile_benchmarks.sh profile cpu \
--repository . \
--benchmark 'IMapJobStorageBenchmark.runningJobGrowth$'
```

Replace `cpu` with `wall`, `lock`, or `gc` as needed. The same command can be used for:

```text
IMapJobStorageBenchmark.completedJobHistoryGrowth$
```

Capture a JFR recording:

```bash
bash tools/benchmarks/profile_benchmarks.sh capture jfr \
--repository . \
--benchmark 'IMapJobStorageBenchmark.runningJobGrowth$'
```

CPU, wall-clock, and lock profiling require `ASYNC_PROFILER_HOME`. The GitHub Actions workflow installs the required profiler automatically.

## Expected outcome

This issue should result in both root-cause analysis and a focused improvement:

- Identify and explain the primary source of the latency variance.
- Improve the benchmark when the fixture or methodology is responsible.
- Optimize the production job-lifecycle storage path when an implementation problem is confirmed.
- Add or update benchmark coverage to validate the improvement.
- Provide before-and-after results collected with the same JDK, runner, benchmark arguments, and storage configuration.
- Compare both operation latency and variance before and after the change.

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with IMapJobStorageBenchmark.runningJobGrowth and completedJobHistoryGrowth, then run the exact methods through the Benchmarks Diagnostics workflow or tools/benchmarks/profile_benchmarks.sh. Compare CPU, wall-clock, lock, GC, or JFR results and inspect the fixture and storage path implicated by profiling. Done means a documented root cause, a focused improvement, updated benchmark coverage, and before-and-after latency and variance results under matching conditions.

Written by the indexing model from the issue text.

Assessment

Tech stack
github-actions, java
Domain
performance, testing
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.