[Improve][Zeta] Investigate and improve intermediate queue benchmark stability
- Dominant language
- Java
- Stars
- 9.7k
- Forks
- 2.4k
- Avg merge
- 3d 9h
- Merged PRs (30d)
- 204
Description
## Description
This is a focused investigation and improvement task for the throughput variance observed in the intermediate queue benchmarks.
It is not a performance regression report and does not attempt to compare the performance of Java 8 with Java 11.
The benchmark results show noticeable sample-to-sample variance within individual runs:
| Benchmark | Java 8 Error / CV | Java 11 Error / CV |
| --- | ---: | ---: |
| `blockingQueueRecordHandoff` | 10.76% / 10.06% | 9.95% / 9.31% |
| `disruptorRecordHandoff` | 22.46% / 21.01% | 18.35% / 17.16% |
Benchmark run:
https://github.com/apache/seatunnel/actions/runs/33723940255
Other benchmarks executed by the same workflow were substantially more stable. The intermediate queue results are therefore worth investigating to determine whether the variance comes from the benchmark fixture, queue implementation, thread scheduling, or execution environment.
The goals of this issue are to:
1. Identify and explain the source of the throughput variance.
2. Improve the benchmark fixture or methodology if it causes the instability.
3. Optimize the production queue path if profiling confirms an implementation bottleneck.
4. Preserve the correctness and lifecycle behavior of both queue implementations.
## Relevant benchmarks
```text
IntermediateQueueBenchmark.blockingQueueRecordHandoff
IntermediateQueueBenchmark.disruptorRecordHandoff
```
## Profiling tools
For an overview of the Zeta benchmark suite and usage, see the [SeaTunnel Zeta Benchmark Guide](https://seatunnel.apache.org/docs/engines/zeta/benchmark).
SeaTunnel provides the `Benchmarks Diagnostics` workflow for running one exact benchmark method with CPU, wall-clock, lock, GC, or JFR profiling:
https://github.com/apache/seatunnel/actions/workflows/benchmarks_diagnostics.yml
The benchmark can also be run directly from the GitHub Actions page by selecting **Benchmarks Diagnostics**, clicking **Run workflow**, and providing the target branch, tag, commit SHA, or trusted PR number together with the exact benchmark method.
## Local profiling
Build the benchmark module:
```bash
./mvnw -Pbenchmark -pl seatunnel-benchmarks -am -DskipTests package
```
Run a profiler locally:
```bash
bash tools/benchmarks/profile_benchmarks.sh profile cpu \
--repository . \
--benchmark 'IntermediateQueueBenchmark.disruptorRecordHandoff$'
```
Replace `cpu` with `wall`, `lock`, or `gc` as needed. The same command can be used for:
```text
IntermediateQueueBenchmark.blockingQueueRecordHandoff$
```
Capture a JFR recording:
```bash
bash tools/benchmarks/profile_benchmarks.sh capture jfr \
--repository . \
--benchmark 'IntermediateQueueBenchmark.disruptorRecordHandoff$'
```
CPU, wall-clock, and lock profiling require `ASYNC_PROFILER_HOME`. The GitHub Actions workflow installs the required profiler automatically.
## Expected outcome
This issue should result in both root-cause analysis and a focused improvement:
- Identify and explain the primary source of the benchmark variance.
- Improve the benchmark when the fixture or methodology is responsible.
- Optimize the production queue path when an implementation problem is confirmed.
- Provide before-and-after results collected with the same JDK, runner, benchmark arguments, and queue parameters.
- Compare both throughput and variance before and after the change.
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.