spring-projects / spring-projects/spring-batch

Spring Batch is very slow when using two datasources (one for batch/task and another for the app)

Open
#4,069 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

has: minimal-example related-to: performance status: waiting-for-reporter type: bug
Dominant language
Java
Stars
3k
Forks
2.5k
Avg merge
6d 53m
Merged PRs (30d)
3

Description

Bug description
Spring batch configured with two datasources (one for batch/task and another for the app) is much slower than using one datasource for both (boot configured or custom). Following the spring recommended approach for two datasources (here and here) does not help. Tried this on the sample file-to-jdbc example provided by spring. The datasources (in my case) point to the same MySql DB server but to different schemas.

Timing observed with the provided input file "data/name-list.csv" in the example:

  1. Default Boot configured datasource - 1 second
  2. One custom datasource (for both Batch/Task and App) - 1 second
  3. Two custom datasources (one each for Batch/Task and App) - 90 seconds

Environment
Java 17, Spring Boot 2.4.x, MySql 8 running on Ubuntu 20.04

Steps to reproduce
Run the file-to-jdbc sample app provided by spring with two data sources created per the recommendations in links above in bug description. More details below on the Minimal complete reproducible section.

Expected behavior
Expect to see the similar timing as one datasource for both batch/task and app

Minimal Complete Reproducible example
I used the sample file-to-jdbc as the baseline and changed it to use two datasources. Modified files attached in zip.
file-to-jdbc-ChangedFiles.zip

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the linked file-to-jdbc sample and the attached file-to-jdbc-ChangedFiles.zip, then reproduce the reported timings with one and two datasources. Compare the Batch/Task and application datasource configurations to identify the source of the slowdown; done means the two-datasource setup no longer takes about 90 seconds and its behavior is verified against the one-datasource cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, mysql
Domain
backend, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.