spring-projects / spring-projects/spring-batch
Spring Batch is very slow when using two datasources (one for batch/task and another for the app)
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
Bug description
Spring batch configured with two datasources (one for batch/task and another for the app) is much slower than using one datasource for both (boot configured or custom). Following the spring recommended approach for two datasources (here and here) does not help. Tried this on the sample file-to-jdbc example provided by spring. The datasources (in my case) point to the same MySql DB server but to different schemas.
Timing observed with the provided input file "data/name-list.csv" in the example:
- Default Boot configured datasource - 1 second
- One custom datasource (for both Batch/Task and App) - 1 second
- Two custom datasources (one each for Batch/Task and App) - 90 seconds
Environment
Java 17, Spring Boot 2.4.x, MySql 8 running on Ubuntu 20.04
Steps to reproduce
Run the file-to-jdbc sample app provided by spring with two data sources created per the recommendations in links above in bug description. More details below on the Minimal complete reproducible section.
Expected behavior
Expect to see the similar timing as one datasource for both batch/task and app
Minimal Complete Reproducible example
I used the sample file-to-jdbc as the baseline and changed it to use two datasources. Modified files attached in zip.
file-to-jdbc-ChangedFiles.zip
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked file-to-jdbc sample and the attached file-to-jdbc-ChangedFiles.zip, then reproduce the reported timings with one and two datasources. Compare the Batch/Task and application datasource configurations to identify the source of the slowdown; done means the two-datasource setup no longer takes about 90 seconds and its behavior is verified against the one-datasource cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, mysql
- Domain
- backend, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100