spring-projects / spring-projects/spring-batch

Number of SQL statements per job launch grows with the number of past executions, and grew further in 6.0

Open
#5,495 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

in: core type: enhancement
Dominant language
Java
Stars
3k
Forks
2.5k
Avg merge
6d 53m
Merged PRs (30d)
3

Description

Launching a job issues a number of SQL statements that grows with how many times the same
job instance has already been executed. This is the concern raised in #4169, but the code
that issue points at (SimpleJobRepository.createJobExecution retrieving the full execution
history) no longer exists, so I measured the current behaviour instead. It is still there on
5.2.2, and about eight times steeper on 6.0.4.

Steps to reproduce

Restart the same job instance 12 times. The step fails on every run so the instance keeps
being restarted with the same identifying parameters. Statements are counted per launch with
datasource-proxy wrapped around the H2 DataSource. The attached projects print the output
below.

Spring Batch 5.2.2:

launch | statements | BATCH_JOB_EXECUTION reads
-------|------------|--------------------------
     1 |         27 |      8
     2 |         35 |     17
     3 |         36 |     18
     4 |         37 |     19
     5 |         38 |     20
     6 |         39 |     21
     7 |         40 |     22
     8 |         41 |     23
     9 |         42 |     24
    10 |         43 |     25
    11 |         44 |     26
    12 |         45 |     27

Spring Batch 6.0.4:

launch | statements | BATCH_JOB_EXECUTION reads
-------|------------|--------------------------
     1 |         32 |     11
     2 |         50 |     21
     3 |         58 |     24
     4 |         66 |     27
     5 |         74 |     30
     6 |         82 |     33
     7 |         90 |     36
     8 |         98 |     39
     9 |        106 |     42
    10 |        114 |     45
    11 |        122 |     48
    12 |        130 |     51

From the second launch onward, each additional past execution costs one more statement on
5.2.2 and eight more on 6.0.4. The entry point is not the cause: on 6.0.4 the numbers are
identical whether the job is launched through JobOperator.start or through the deprecated
TaskExecutorJobLauncher, so the growth comes from the repository layer rather than from
the newer API.

Which statements repeat

On the twelfth launch, with 13 executions on the instance:

5.2.2
13x  SELECT JOB_EXECUTION_ID, PARAMETER_NAME, ... FROM BATCH_JOB_EXECUTION_PARAMS WHERE JOB_EXECUTION_ID = ?
 4x  SELECT VERSION FROM BATCH_JOB_EXECUTION WHERE JOB_EXECUTION_ID=?
 3x  SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_JOB_EXECUTION_CONTEXT WHERE JOB_EXECUTION_ID = ?

6.0.4
13x  SELECT JOB_EXECUTION_ID, PARAMETER_NAME, ... FROM BATCH_JOB_EXECUTION_PARAMS WHERE JOB_EXECUTION_ID = ?
13x  SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_JOB_EXECUTION_CONTEXT WHERE JOB_EXECUTION_ID = ?
13x  SELECT SHORT_CONTEXT, SERIALIZED_CONTEXT FROM BATCH_STEP_EXECUTION_CONTEXT WHERE STEP_EXECUTION_ID = ?
12x  SELECT JI.JOB_INSTANCE_ID FROM BATCH_JOB_INSTANCE JI, BATCH_JOB_EXECUTION JE WHERE JOB_EXECUTION_ID = ? AND ...
12x  SELECT JOB_INSTANCE_ID, JOB_NAME, JOB_KEY, VERSION FROM BATCH_JOB_INSTANCE WHERE JOB_INSTANCE_ID = ?

On both versions, every execution in the history costs one additional query for its job
parameters. On 6.0.4 the job execution context, the step execution context and the job
instance are each read once per execution as well, which is where the steeper slope comes
from.

Minimal Complete Reproducible example

spring-batch-4169-mcve.zip

Attached are two self-contained Maven projects, one per version:

mvn package exec:java -Dexec.mainClass=repro.Repro

One note on the 6.0 project: @EnableBatchProcessing alone now configures a resourceless
job repository, so the reproduction needs @EnableJdbcJobRepository as well. Without it no
SQL is issued at all, which is easy to mistake for the problem being gone.

Relation to #4169

Same underlying concern, different code. I did not comment there because that issue is
anchored to a method that has since been rewritten, and the numbers above are new. Happy for
you to link them or close whichever you prefer.

Environment

Spring Batch 5.2.2 and 6.0.4, JDK 17 (Temurin 17.0.20), H2 2.2.224 in-memory, macOS.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Run the attached Maven reproductions with mvn package exec:java -Dexec.mainClass=repro.Repro, comparing the Spring Batch 5.2.2 and 6.0.4 results. Trace the repository-layer calls around SimpleJobRepository.createJobExecution, JobOperator.start, and TaskExecutorJobLauncher, then verify that repeated launches no longer produce SQL growth proportional to past executions.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring, sql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
52/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.