spring-projects / spring-projects/spring-batch
Built in support for partitioning of multi-step flows instead of single steps
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
Expected Behavior
Built in support or tags for full flows to be partitioned.
Example: a flow F consists of 5 steps s1-> s2 -> s3 -> s4 -> s5
Support for flow F to be portioned i.e. multiple instances of flow F ( s1-> s2 -> s3 -> s4 -> s5 ) could run in parallel through portioning.
Current Behavior
No specified tags to execute entire flows in parallel, although single steps partitioning can be done, there is no such support for portioning a flow of multiple steps.
Context
How has this issue affected you? My application required a flow consisting of multiple steps to be portioned, which could not be accomplished directly
What are you trying to accomplish? A batch job where each flow, consisting of series of steps, to be partitioned using Manager-Worker pattern
What other alternatives have you considered? I tried to use create custom portioning on a flow instead of step, but that did not work, and also caused several data anomalies due to sharing of resources.
Are you aware of any workarounds? Not that i am aware of
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by examining the existing single-step partitioning and custom partitioning behavior described in the issue, along with how a multi-step flow is executed. Define how multiple complete flow instances can run in parallel under the Manager-Worker pattern, and verify that the result avoids the reported shared-resource data anomalies.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- backend, data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 28/100