spring-projects / spring-projects/spring-batch

Improve documentation with regard to chunk scanning

Open
#3,946 5 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

in: documentation type: enhancement
Dominant language
Java
Stars
3k
Forks
2.5k
Avg merge
6d 53m
Merged PRs (30d)
3

Description

Bug description
A simple faultTolerant step configuration using NeverRetryPolicy and AlwaysSkipItemSkipPolicy always retry the items when any exception occurs. Unless I am missing something, I would expect to do not see any retries at all.

Environment
Spring Batch 4.3.3
jdk 15.0.2
Postgres 12.4

Steps to reproduce

  • Configure a faultTolerent step with policies: NeverRetryPolicy and AlwaysSkipItemSkipPolicy
  • Configure the writer to throw a RuntimeException
  • Launch a job with that step
  • Check number of times that the writer is called

Expected behavior
With 5 items to be processed, I would expect that the service would be called only 5 times.

Minimal Complete Reproducible example

Job Configuration:

@Configuration
@EnableBatchProcessing
@RequiredArgsConstructor
@Slf4j
public class ChunkSizeBatchConfig {

  private final JobBuilderFactory jobBuilderFactory;
  private final StepBuilderFactory stepBuilderFactory;
  private final MockedService mockedService;

  @Bean
  Job chunkSizeJob() {

    var itemReader = new ListItemReader<>(List.of(1, 2, 3, 4, 5));

    var itemWriter = new ItemWriter<Integer>() {
      @Override
      public void write(@Nonnull List<? extends Integer> items) {
        logger.info("chunkSizeJob processing item \"{}\"", items);
        items.forEach(mockedService::doSomething);
      }
    };

    var step = stepBuilderFactory.get("testStep")
        .<Integer, Integer>chunk(1)
        .reader(itemReader)
        .writer(itemWriter)
        .faultTolerant()
        .skipPolicy(new AlwaysSkipItemSkipPolicy())
        .retryPolicy(new NeverRetryPolicy())
        .build();

    return jobBuilderFactory.get("testJob")
        .incrementer(new RunIdIncrementer())
        .flow(step)
        .end()
        .build();
  }

  interface MockedService {
    void doSomething(int obj);
  }

}

Test class

@SpringBatchTest
@DataJpaTest(excludeAutoConfiguration = {TestDatabaseAutoConfiguration.class})
@EnableAutoConfiguration
@EntityScan("com.possiblefinance.ffp.adapter.out.persistence")
// overrides @SpringBatchTest @TestExecutionListeners
@TestExecutionListeners(
    listeners = {DBRiderTestExecutionListener.class, StepScopeTestExecutionListener.class, JobScopeTestExecutionListener.class},
    mergeMode = TestExecutionListeners.MergeMode.MERGE_WITH_DEFAULTS
)
// Spring Batch needs to open it's own transaction for the test
@Transactional(propagation = Propagation.SUPPORTS)
@ContextConfiguration(classes = {ChunkSizeBatchConfig.class})
class ChunkSizeBatchConfigIT {

  @Autowired
  protected JobLauncherTestUtils jobLauncherTestUtils;

  @MockBean
  private ChunkSizeBatchConfig.MockedService mockedService;

  @Test
  @SneakyThrows
  void testJobCheckCompletedEvenWithException() {
    var jobParameters = new JobParametersBuilder()
        .addLong("testExecutionKey", System.currentTimeMillis())
        .addString(BatchConstants.JOB_PARAMETER_PARTITIONING_KEYS, "1,2,3")
        .toJobParameters();

    Mockito.doThrow(new RuntimeException("Any error"))
        .when(mockedService).doSomething(anyInt());

    var jobExecution1 = jobLauncherTestUtils.launchJob(jobParameters);
    assertThat(jobExecution1.getExitStatus().getExitCode()).isEqualTo("COMPLETED");

    verify(mockedService, times(5)).doSomething(anyInt());
  }

}

Test Output:

2021-06-18 16:13:04.462  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[1]"
2021-06-18 16:13:04.469  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[1]"
2021-06-18 16:13:04.477  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[2]"
2021-06-18 16:13:04.478  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[2]"
2021-06-18 16:13:04.485  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[3]"
2021-06-18 16:13:04.486  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[3]"
2021-06-18 16:13:04.493  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[4]"
2021-06-18 16:13:04.494  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[4]"
2021-06-18 16:13:04.502  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[5]"
2021-06-18 16:13:04.503  INFO 11727 --- [           main] c.p.f.a.j.configs.ChunkSizeBatchConfig   : chunkSizeJob processing item "[5]"
2021-06-18 16:13:04.522  INFO 11727 --- [           main] o.s.batch.core.step.AbstractStep         : Step: [testStep] executed in 79ms
2021-06-18 16:13:04.539  INFO 11727 --- [           main] o.s.b.c.l.support.SimpleJobLauncher      : Job: [FlowJob: [name=testJob]] completed with the following parameters: [{testExecutionKey=1624032784328, partitioningKeys=1,2,3}] and the following status: [COMPLETED] in 115ms

org.mockito.exceptions.verification.TooManyActualInvocations: 
com.possiblefinance.ffp.application.jobs.configs.ChunkSizeBatchConfig$MockedService#0 bean.doSomething(
    <any integer>
);
Wanted 5 times:
-> at com.possiblefinance.ffp.application.jobs.configs.ChunkSizeBatchConfigIT.testJobCheckCompletedEvenWithException(ChunkSizeBatchConfigIT.java:52)
But was 10 times:
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)
-> at java.base/java.util.ArrayList.forEach(ArrayList.java:1511)
-> at java.base/java.util.Collections$SingletonList.forEach(Collections.java:4933)

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the ChunkSizeBatchConfig and ChunkSizeBatchConfigIT examples in the issue, then reproduce the fault-tolerant step with five items and the stated retry and skip policies. Compare the expected five service calls with the reported ten and determine whether the behavior or its documentation needs correction.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spring
Domain
backend
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.