spring-projects / spring-projects/spring-batch
Number of SQL statements per job launch grows with the number of past executions, and grew further in 6.0
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
Launching a job issues a number of SQL statements that grows with how many times the same
job instance has already been executed. This is the concern raised in #4169, but the code
that issue points at (SimpleJobRepository.createJobExecution retrieving the full execution
history) no longer exists, so I measured the current behaviour instead. It is still there on
5.2.2, and about eight times steeper on 6.0.4.
Steps to reproduce
Restart the same job instance 12 times. The step fails on every run so the instance keeps
being restarted with the same identifying parameters. Statements are counted per launch with
datasource-proxy wrapped around the H2 DataSource. The attached projects print the output
below.
Spring Batch 5.2.2:
launch | statements | BATCH_JOB_EXECUTION reads
-------|------------|--------------------------
1 | 27 | 8
2 | 35 | 17
3 | 36 | 18
4 | 37 | 19
5 | 38 | 20
6 | 39 | 21
7 | 40 | 22
8 | 41 | 23
9 | 42 | 24
10 | 43 | 25
11 | 44 | 26
12 | 45 | 27
Spring Batch 6.0.4:
launch | statements | BATCH_JOB_EXECUTION reads
-------|------------|--------------------------
1 | 32 | 11
2 | 50 | 21
3 | 58 | 24
4 | 66 | 27
5 | 74 | 30
6 | 82 | 33
7 | 90 | 36
8 | 98 | 39
9 | 106 | 42
10 | 114 | 45
11 | 122 | 48
12 | 130 | 51
From the second launch onward, each additional past execution costs one more statement on
5.2.2 and eight more on 6.0.4. The entry point is not the cause: on 6.0.4 the numbers are
identical whether the job is launched through JobOperator.start or through the deprecated
TaskExecutorJobLauncher, so the growth comes from the repository layer rather than from
the newer API.
Which statements repeat
On the twelfth launch, with 13 executions on the instance:
5.2.2
13x SELECT JOB_EXECUTION_ID, PARAMETER_NAME, ... FROM BATCH_JOB_EXECUTION_PARAMS WHERE JOB_EXECUTION_ID = ?
4x SELECT VERSION FROM BATCH_JOB_EXECUTION WHERE JOB_EXECUTION_ID=?
3x SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_JOB_EXECUTION_CONTEXT WHERE JOB_EXECUTION_ID = ?
6.0.4
13x SELECT JOB_EXECUTION_ID, PARAMETER_NAME, ... FROM BATCH_JOB_EXECUTION_PARAMS WHERE JOB_EXECUTION_ID = ?
13x SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_JOB_EXECUTION_CONTEXT WHERE JOB_EXECUTION_ID = ?
13x SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_STEP_EXECUTION_CONTEXT WHERE STEP_EXECUTION_ID = ?
12x SELECT JI.JOB_INSTANCE_ID FROM BATCH_JOB_INSTANCE JI, BATCH_JOB_EXECUTION JE WHERE JOB_EXECUTION_ID = ? AND ...
12x SELECT JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION FROM BATCH_JOB_INSTANCE WHERE JOB_INSTANCE_ID = ?
On both versions, every execution in the history costs one additional query for its job
parameters. On 6.0.4 the job execution context, the step execution context and the job
instance are each read once per execution as well, which is where the steeper slope comes
from.
Minimal Complete Reproducible example
Attached are two self-contained Maven projects, one per version:
mvn package exec:java -Dexec.mainClass=repro.Repro
One note on the 6.0 project: @EnableBatchProcessing alone now configures a resourceless
job repository, so the reproduction needs @EnableJdbcJobRepository as well. Without it no
SQL is issued at all, which is easy to mistake for the problem being gone.
Relation to #4169
Same underlying concern, different code. I did not comment there because that issue is
anchored to a method that has since been rewritten, and the numbers above are new. Happy for
you to link them or close whichever you prefer.
Environment
Spring Batch 5.2.2 and 6.0.4, JDK 17 (Temurin 17.0.20), H2 2.2.224 in-memory, macOS.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Run the attached Maven reproductions with mvn package exec:java -Dexec.mainClass=repro.Repro, comparing the Spring Batch 5.2.2 and 6.0.4 results. Trace the repository-layer calls around SimpleJobRepository.createJobExecution, JobOperator.start, and TaskExecutorJobLauncher, then verify that repeated launches no longer produce SQL growth proportional to past executions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring, sql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 52/100