spring-projects / spring-projects/spring-batch

Built in support for partitioning of multi-step flows instead of single steps

Open
#4,369 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

status: waiting-for-reporter type: feature
Dominant language
Java
Stars
3k
Forks
2.5k
Avg merge
6d 53m
Merged PRs (30d)
3

Description

Expected Behavior

Built in support or tags for full flows to be partitioned.

Example: a flow F consists of 5 steps s1-> s2 -> s3 -> s4 -> s5
Support for flow F to be portioned i.e. multiple instances of flow F ( s1-> s2 -> s3 -> s4 -> s5 ) could run in parallel through portioning.

Current Behavior

No specified tags to execute entire flows in parallel, although single steps partitioning can be done, there is no such support for portioning a flow of multiple steps.

Context

How has this issue affected you? My application required a flow consisting of multiple steps to be portioned, which could not be accomplished directly
What are you trying to accomplish? A batch job where each flow, consisting of series of steps, to be partitioned using Manager-Worker pattern
What other alternatives have you considered? I tried to use create custom portioning on a flow instead of step, but that did not work, and also caused several data anomalies due to sharing of resources.

Are you aware of any workarounds? Not that i am aware of

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by examining the existing single-step partitioning and custom partitioning behavior described in the issue, along with how a multi-step flow is executed. Define how multiple complete flow instances can run in parallel under the Manager-Worker pattern, and verify that the result avoids the reported shared-resource data anomalies.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.