Maintain compatibility with Avro 1.11
- Dominant language
- Java
- Stars
- 9.2k
- Forks
- 3.5k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 132
Description
I'm running Iceberg reads via [Apache Beam](https://github.com/apache/beam)'s iceberg [extension](https://github.com/apache/beam/tree/master/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg). Beam, and my project, are on Avro 1.11. When executing an Iceberg read, I hit the following runtime exception:
```
Exception in thread "sbt-bg-threads-1" java.lang.NoSuchMethodError: 'org.apache.avro.LogicalTypes$TimestampNanos org.apache.avro.LogicalTypes.timestampNanos()'
at org.apache.iceberg.avro.TypeToSchema.(TypeToSchema.java:50)
at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:64)
at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:59)
at org.apache.iceberg.GenericManifestFile.(GenericManifestFile.java:42)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.apache.iceberg.common.DynConstructors$Ctor.newInstanceChecked(DynConstructors.java:51)
at org.apache.iceberg.common.DynConstructors$Ctor.newInstance(DynConstructors.java:64)
at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:67)
at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:42)
at org.apache.iceberg.avro.ValueReaders$PlannedStructReader.read(ValueReaders.java:1025)
at org.apache.iceberg.avro.InternalReader.read(InternalReader.java:107)
at org.apache.iceberg.avro.NameMappingDatumReader.read(NameMappingDatumReader.java:57)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:263)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:248)
at org.apache.iceberg.io.CloseableIterator$1.next(CloseableIterator.java:55)
at org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:370)
at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:332)
at org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:261)
at org.apache.iceberg.ManifestLists.read(ManifestLists.java:42)
at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:185)
at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:211)
at org.apache.iceberg.DataTableScan.doPlanFiles(DataTableScan.java:68)
at org.apache.iceberg.SnapshotScan.planFiles(SnapshotScan.java:139)
at org.apache.beam.sdk.io.iceberg.ScanSource.wholeTableReadTask(ScanSource.java:73)
...
```
This is happening because `timestamp-nanos` wasn't added to Avro until 1.12 (compare [1.11](https://github.com/apache/avro/blob/release-1.11.5/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L187-L197) types vs [1.12](https://github.com/apache/avro/blob/release-1.12.1/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L196-L208)).
Upgrading Avro versions can be a nontrivial operation for various reasons - I was wondering if Iceberg would accept a change that gates the static initialization of `TIMESTAMP_NANO_SCHEMA` and `TIMESTAMPTZ_NANO_SCHEMA` behind an Avro version check. There's precedent for this in [parquet-java](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.1/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L264-L266), however I'm not sure if Iceberg has made any commitments about Avro backwards-compatibility/if it's something that would be painful to maintain in the future.
Let me know what you think; I have a proof of concept [here](https://github.com/apache/iceberg/compare/main...clairemcginty:iceberg:avro-1.11-schema-compat?expand=1) that works. thanks!
Contributor guide
Research direction
Start with the static initialization of TIMESTAMP_NANO_SCHEMA and TIMESTAMPTZ_NANO_SCHEMA in TypeToSchema.java, then review the AvroSchemaUtil conversion path and the linked proof of concept. Compare the compatibility approach with parquet-java's AvroRecordConverter.java precedent and verify that Iceberg reads work with Avro 1.11 while newer Avro versions retain nanosecond timestamp support.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100