GcsUtil can try to execute empty batches in some cases.
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
I think, but have not confirmed, that this happens because we try to remove an empty set of files.
```
SEVERE: 2016-12-22T16:59:17.063Z: (e72c8b756ce69f1a): java.io.IOException: Error executing batch GCS
request
at org.apache.beam.sdk.util.GcsUtil.executeBatches(GcsUtil.java:486)
at org.apache.beam.sdk.util.GcsUtil.remove(GcsUtil.java:535)
at org.apache.beam.sdk.util.GcsIOChannelFactory.remove(GcsIOChannelFactory.java:108)
at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:468)
at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:382)
at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
Caused by: java.util.concurrent.ExecutionException:
java.lang.IllegalArgumentException: no JSON input found
at org.apache.beam.sdk.repackaged.com.google.common.util.concurrent.AbstractFuture.getDoneValue(AbstractFuture.java:476)
at org.apache.beam.sdk.repackaged.com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:455)
at org.apache.beam.sdk.repackaged.com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:79)
at org.apache.beam.sdk.util.GcsUtil.executeBatches(GcsUtil.java:481)
at org.apache.beam.sdk.util.GcsUtil.remove(GcsUtil.java:535)
at org.apache.beam.sdk.util.GcsIOChannelFactory.remove(GcsIOChannelFactory.java:108)
at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.removeTemporaryFiles(FileBasedSink.java:468)
at org.apache.beam.sdk.io.FileBasedSink$FileBasedWriteOperation.finalize(FileBasedSink.java:382)
at org.apache.beam.sdk.io.Write$Bound$2.processElement(Write.java:417)
at org.apache.beam.sdk.io.Write$Bound$2$auxiliary$3HTfRVcv.invokeProcessElement(Unknown
Source)
at org.apache.beam.sdk.transforms.DoFnAdapters$SimpleDoFnAdapter.processElement(DoFnAdapters.java:234)
at org.apache.beam.runners.core.SimpleOldDoFnRunner.invokeProcessElement(SimpleOldDoFnRunner.java:118)
at org.apache.beam.runners.core.SimpleOldDoFnRunner.processElement(SimpleOldDoFnRunner.java:104)
at com.google.cloud.dataflow.worker.runners.worker.SimpleParDoFn.processElement(SimpleParDoFn.java:192)
```
Imported from Jira [BEAM-1209](https://issues.apache.org/jira/browse/BEAM-1209). Original Jira may contain additional context.
Reported by: dhalperi.
Contributor guide
Research direction
Start in GcsUtil.java at executeBatches and remove, using the stack trace to follow the cleanup path through GcsIOChannelFactory.java and FileBasedSink.java. Confirm how an empty removal set reaches batch execution; done means empty batches no longer trigger the reported GCS error, with regression coverage for that path.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- gcp, java
- Domain
- cloud
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100