Using BigQueryIO.read with DIRECT_READ causes Illegal Mutation
- Dominant language
- Java
- Stars
- 8.7k
- Forks
- 4.7k
- Avg merge
- 1d 20h
- Merged PRs (30d)
- 196
Description
Code to read from BigQuery that is causing the issue:
```
pipeline
.apply(BigQueryIO
.read(SchemaAndRecord::getRecord)
.from(options.getTableRef())
.withMethod(Method.DIRECT_READ)
.withCoder(AvroCoder.of(schema)))
```
If we remove .withMethod(Method.DIRECT_READ) then there is no issue.
The error is:
```
org.apache.beam.sdk.util.IllegalMutationException: PTransform BigQueryIO.TypedRead/Read(BigQueryStorageTableSource)
mutated value {"device_id": "rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time":
1564412307969368, "humidity": 74.3} after it was output (new value was {"device_id": "rpi-rpi0-thermostat",
"temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}). Values
must not be mutated in any way after being output.
at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
(ImmutabilityCheckingBundleFactory.java:134)
at org.apache.beam.runners.direct.EvaluationContext.commitBundles
(EvaluationContext.java:210)
at org.apache.beam.runners.direct.EvaluationContext.handleResult (EvaluationContext.java:151)
at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult (QuiescenceDriver.java:262)
at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle (DirectTransformExecutor.java:189)
at org.apache.beam.runners.direct.DirectTransformExecutor.run (DirectTransformExecutor.java:126)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run
(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run
(Thread.java:748)
Caused by: org.apache.beam.sdk.util.IllegalMutationException: Value {"device_id":
"rpi-rpi0-thermostat", "temperature_c": 20.0, "temperature_f": 52.0, "sample_time": 1564412307969368,
"humidity": 74.3} mutated illegally, new value was {"device_id": "rpi-rpi0-thermostat", "temperature_c":
20.0, "temperature_f": 52.0, "sample_time": 1564412360458615, "humidity": 74.7}. Encoding was AiZycGktcnBpMC10aGVybW9zdGF0AgAAAAAAADRAAgAAAAAAAEpAArDVsP7jtMcFAjMzMzMzk1JA,
now AiZycGktcnBpMC10aGVybW9zdGF0AgAAAAAAADRAAgAAAAAAAEpAAu6FuLDktMcFAs3MzMzMrFJA.
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.illegalMutation
(MutationDetectors.java:153)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodifiedThrowingCheckedExceptions
(MutationDetectors.java:148)
at org.apache.beam.sdk.util.MutationDetectors$CodedValueMutationDetector.verifyUnmodified
(MutationDetectors.java:123)
at org.apache.beam.runners.direct.ImmutabilityCheckingBundleFactory$ImmutabilityEnforcingBundle.commit
(ImmutabilityCheckingBundleFactory.java:124)
at org.apache.beam.runners.direct.EvaluationContext.commitBundles
(EvaluationContext.java:210)
at org.apache.beam.runners.direct.EvaluationContext.handleResult (EvaluationContext.java:151)
at org.apache.beam.runners.direct.QuiescenceDriver$TimerIterableCompletionCallback.handleResult (QuiescenceDriver.java:262)
at org.apache.beam.runners.direct.DirectTransformExecutor.finishBundle (DirectTransformExecutor.java:189)
at org.apache.beam.runners.direct.DirectTransformExecutor.run (DirectTransformExecutor.java:126)
at java.util.concurrent.Executors$RunnableAdapter.call (Executors.java:511)
at java.util.concurrent.FutureTask.run
(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker (ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run (ThreadPoolExecutor.java:624)
at java.lang.Thread.run
(Thread.java:748)
```
Imported from Jira [BEAM-8029](https://issues.apache.org/jira/browse/BEAM-8029). Original Jira may contain additional context.
Reported by: clarsen.
Contributor guide
Assessment
This issue has not been assessed yet.