spring-projects / spring-projects/spring-batch
Improve documentation with regard to chunk scanning
Nobody has claimed this yet.
- Dominant language
- Java
- Stars
- 3k
- Forks
- 2.5k
- Avg merge
- 6d 53m
- Merged PRs (30d)
- 3
Description
Bug description
A simple faultTolerant step configuration using NeverRetryPolicy and AlwaysSkipItemSkipPolicy always retry the items when any exception occurs. Unless I am missing something, I would expect to do not see any retries at all.
Environment
Spring Batch 4.3.3
jdk 15.0.2
Postgres 12.4
Steps to reproduce
- Configure a faultTolerent step with policies:
NeverRetryPolicyandAlwaysSkipItemSkipPolicy - Configure the writer to throw a RuntimeException
- Launch a job with that step
- Check number of times that the writer is called
Expected behavior
With 5 items to be processed, I would expect that the service would be called only 5 times.
Minimal Complete Reproducible example
Job Configuration:
@Configuration
@EnableBatchProcessing
@RequiredArgsConstructor
@Slf4j
public class ChunkSizeBatchConfig {
private final JobBuilderFactory jobBuilderFactory;
private final StepBuilderFactory stepBuilderFactory;
private final MockedService mockedService;
@Bean
Job chunkSizeJob() {
var itemReader = new ListItemReader<>(List.of(1, 2, 3, 4, 5));
var itemWriter = new ItemWriter<Integer>() {
@Override
public void write(@Nonnull List<? extends Integer> items) {
logger.info("chunkSizeJob processing item \"{}\"", items);
items.forEach(mockedService::doSomething);
}
};
var step = stepBuilderFactory.get("testStep")
.<Integer, Integer>chunk(1)
.reader(itemReader)
.writer(itemWriter)
.faultTolerant()
.skipPolicy(new AlwaysSkipItemSkipPolicy())
.retryPolicy(new NeverRetryPolicy())
.build();
return jobBuilderFactory.get("testJob")
.incrementer(new RunIdIncrementer())
.flow(step)
.end()
.build();
}
interface MockedService {
void doSomething(int obj);
}
}
Test class
@SpringBatchTest
@DataJpaTest(excludeAutoConfiguration = {TestDatabaseAutoConfiguration.class})
@EnableAutoConfiguration
@EntityScan("com.possiblefinance.ffp.adapter.out.persistence")
// overrides @SpringBatchTest @TestExecutionListeners
@TestExecutionListeners(
listeners = {DBRiderTestExecutionListener.class, StepScopeTestExecutionListener.class, JobScopeTestExecutionListener.class},
mergeMode = TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS
)
// Spring Batch needs to open it's own transaction for the test
@Transactional(propagation = Propagation.SUPPORTS)
@ContextConfiguration(classes = {ChunkSizeBatchConfig.class})
class ChunkSizeBatchConfigIT {
@Autowired
protected JobLauncherTestUtils jobLauncherTestUtils;
@MockBean
private ChunkSizeBatchConfig.MockedService mockedService;
@Test
@SneakyThrows
void testJobCheckCompletedEvenWithException() {
var jobParameters = new JobParametersBuilder()
.addLong("testExecutionKey", System.currentTimeMillis())
.addString(BatchConstants.JOB_PARAMETER_PARTITIONING_KEYS, "1,2,3")
.toJobParameters();
Mockito.doThrow(new RuntimeException("Any error"))
.when(mockedService).doSomething(anyInt());
var jobExecution1 = jobLauncherTestUtils.launchJob(jobParameters);
assertThat(jobExecution1.getExitStatus().getExitCode()).isEqualTo("COMPLETED");
verify(mockedService, times(5)).doSomething(anyInt());
}
}
Test Output:
2021-06-18 16:13:04.462 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[1]"
2021-06-18 16:13:04.469 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[1]"
2021-06-18 16:13:04.477 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[2]"
2021-06-18 16:13:04.478 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[2]"
2021-06-18 16:13:04.485 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[3]"
2021-06-18 16:13:04.486 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[3]"
2021-06-18 16:13:04.493 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[4]"
2021-06-18 16:13:04.494 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[4]"
2021-06-18 16:13:04.502 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[5]"
2021-06-18 16:13:04.503 INFO 11727 --- [ main] c.p.f.a.j.configs.ChunkSizeBatchConfig : chunkSizeJob processing item "[5]"
2021-06-18 16:13:04.522 INFO 11727 --- [ main] o.s.batch.core.step.AbstractStep : Step: [testStep] executed in 79ms
2021-06-18 16:13:04.539 INFO 11727 --- [ main] o.s.b.c.l.support.SimpleJobLauncher : Job: [FlowJob: [name=testJob]] completed with the following parameters: [{testExecutionKey=1624032784328, partitioningKeys=1,2,3}] and the following status: [COMPLETED] in 115ms
org.mockito.exceptions.verification.TooManyActualInvocations:
com.possiblefinance.ffp.application.jobs.configs.ChunkSizeBatchConfig$MockedService#0 bean.doSomething(
<any integer>
);
Wanted 5 times:
-> at com.possiblefinance.ffp.application.jobs.configs.ChunkSizeBatchConfigIT.testJobCheckCompletedEvenWithException(ChunkSizeBatchConfigIT.java:52)
But was 10 times:
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the ChunkSizeBatchConfig and ChunkSizeBatchConfigIT examples in the issue, then reproduce the fault-tolerant step with five items and the stated retry and skip policies. Compare the expected five service calls with the reported ten and determine whether the behavior or its documentation needs correction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spring
- Domain
- backend
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100