apache / apache/beam

SyntheticSource option forceNumInitialBundles is not optional in all cases

Open
#19,405 0 comments 0 reactions 0 assignees View on GitHub
bug P3 tests
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

It fails when numRecords is between 1 and  3.

I noticed it by running SyntheticDataPublisher:
```

./gradlew :beam-sdks-java-load-tests:run -PloadTest.args='--pubSubTopic=projects/apache-beam-io-testing/topics/loadtests 
--sourceOptions={"numRecords":2}' -PloadTest.mainClass="org.apache.beam.sdk.loadtests.SyntheticDataPublisher"
```

 

The exception:

 
```

Exception in thread "main" java.lang.OutOfMemoryError: Requested array size exceeds VM limit
     
  at org.apache.beam.sdk.io.synthetic.BundleSplitter.getRelativeBundleSizes(BundleSplitter.java:60)
 
      at org.apache.beam.sdk.io.synthetic.BundleSplitter.getBundleSizes(BundleSplitter.java:39)
   
    at org.apache.beam.sdk.io.synthetic.SyntheticBoundedSource.split(SyntheticBoundedSource.java:146)
 
      at org.apache.beam.runners.direct.BoundedReadEvaluatorFactory$InputProvider.getInitialInputs(BoundedReadEvaluatorFactory.java:214)
 
      at org.apache.beam.runners.direct.ReadEvaluatorFactory$InputProvider.getInitialInputs(ReadEvaluatorFactory.java:89)
 
      at org.apache.beam.runners.direct.RootProviderRegistry.getInitialInputs(RootProviderRegistry.java:76)
 
      at org.apache.beam.runners.direct.ExecutorServiceParallelExecutor.start(ExecutorServiceParallelExecutor.java:155)
 
      at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:208)
        at org.apache.beam.runners.direct.DirectRunner.run(DirectRunner.java:67)
 
      at org.apache.beam.sdk.Pipeline.run(Pipeline.java:313)
        at org.apache.beam.sdk.Pipeline.run(Pipeline.java:299)
 
      at org.apache.beam.sdk.loadtests.SyntheticDataPublisher.main(SyntheticDataPublisher.java:150)

```

 

Imported from Jira [BEAM-7497](https://issues.apache.org/jira/browse/BEAM-7497). Original Jira may contain additional context.
Reported by: kasiak.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.