spring-projects / spring-projects/spring-batch
Enhance RedisItemReader/Writer performance using Redis pipelining
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
RedisItemReader and RedisItemWriter currently process items one by one, which leads to excessive network I/O and causes performance bottlenecks in high-volume jobs.
I propose using Redis pipelining, which bundles multiple commands into a single round-trip.
Below are the results of the Redis benchmark tests on my iMac.
(Environment = Redis cluster via Testcontainers on a local network, chunk size = 1,000)
| Task | 1k items | 10k items | 100k items | 1M items |
|---|---|---|---|---|
| read | 0.845 s | 8.354 s | 80.610 s | 1218.773 s |
| read (Pipeline) | 0.204 s | 1.244 s | 8.733 s | 97.682 s |
| write | 0.858 s | 8.598 s | 79.786 s | 1047.780 s |
| write (Pipeline) | 0.674 s | 0.725 s | 1.726 s | 16.653 s |
| delete | 0.846 s | 8.176 s | 79.777 s | 855.710 s |
| delete (Pipeline) | 0.100 s | 0.160 s | 1.379 s | 13.524 s |
According to the official Redis documentation, pipelining can deliver roughly a 10× performance enhancement, and my benchmarks showed similar results.
For the curious, I’ve put my benchmark source code on my GitHub branch : https://github.com/noojung/spring-batch/commit/d3f48420e60869a6e15ff845ebbe736e7f2f0df8
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Locate RedisItemReader and RedisItemWriter, then review how they currently process items one at a time. Use the linked benchmark source and Redis pipelining documentation as context. Done means the reader and writer use pipelined Redis operations while preserving their existing behavior and improving high-volume benchmark results.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, redis
- Domain
- backend, databases
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100