apache / apache/beam

[Bug]: Non-Option Avro Union with generated SpecificRecord

Open
#22,574 2 comments 0 reactions 0 assignees View on GitHub
bug core java P3
Dominant language
Java
Stars
8.7k
Forks
4.7k
Avg merge
1d 20h
Merged PRs (30d)
196

Description

### What happened?

Hello, I am new to beam so I am probably doing something wrong:

I am trying to create PCollections of a SpecificRecord which can convert between Rows and the java generated class.

For fields with unions that are non-optional, I am hitting exceptiosn:

Here is a test avro schema:
```
{
"name": "MultiUnionSpecificRecord",
"namespace": "specificrecord",
"type": "record",
"doc": "",
"fields": [
{
"name": "foo",
"type": [ "int", "string" ],
"doc": ""
}
]
}
```

Using this to generate the MultiUnionSpecificRecord class:
```
java -jar .avro-tools-1.9.0.jar compile -string -dateTimeLogicalTypeImpl joda schema \
src/test/resources/MultiUnionSpecificRecord.avsc \
src/test/java/specificrecord/
```

Running this test:
```
MultiUnionSpecificRecord record = MultiUnionSpecificRecord
.newBuilder()
.setFoo(0)
.build();

Pipeline p = TestPipeline.create().enableAbandonedNodeEnforcement(false);

PCollection events = p.apply(
Create.of(record)
.withSchema(
AvroUtils.toBeamSchema(specificrecord.MultiUnionSpecificRecord.getClassSchema()),
TypeDescriptor.of(specificrecord.MultiUnionSpecificRecord.class),
AvroUtils.getToRowFunction(specificrecord.MultiUnionSpecificRecord.class, specificrecord.MultiUnionSpecificRecord.getClassSchema()),
AvroUtils.getFromRowFunction(specificrecord.MultiUnionSpecificRecord.class)
)
);
System.out.println("Events Schema: " + events.getSchema());
p.run();
```

Hitting this exception:
```
class java.lang.String cannot be cast to class org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value (java.lang.String is in module java.base of loader 'bootstrap'; org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value is in unnamed module of loader 'app')
java.lang.ClassCastException: class java.lang.String cannot be cast to class org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value (java.lang.String is in module java.base of loader 'bootstrap'; org.apache.beam.sdk.schemas.logicaltypes.OneOfType$Value is in unnamed module of loader 'app')
at org.apache.beam.sdk.schemas.GetterBasedSchemaProvider$RowValueGettersFactory$GetOneOf.convert(GetterBasedSchemaProvider.java:311)
at org.apache.beam.sdk.schemas.GetterBasedSchemaProvider$RowValueGettersFactory$Converter.get(GetterBasedSchemaProvider.java:346)
at org.apache.beam.sdk.values.RowWithGetters.getValue(RowWithGetters.java:71)
at org.apache.beam.sdk.coders.RowCoderGenerator$EncodeInstruction.encodeDelegate(RowCoderGenerator.java:325)
at org.apache.beam.sdk.coders.Coder$ByteBuddy$ZBaqkdOF.encode(Unknown Source
```

Relevant versions:
Java 11
```
org.apache.beam:beam-sdks-java-core:2.40.0
...
org.apache.avro:avro:1.8.2 -> 1.9.2
```

### Issue Priority

Priority: 3

### Issue Component

Component: sdk-java-core

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.