apache / apache/iceberg

Maintain compatibility with Avro 1.11

Open
#16,781 1 comment 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
9.2k
Forks
3.5k
Avg merge
2d 11h
Merged PRs (30d)
132

Description

I'm running Iceberg reads via [Apache Beam](https://github.com/apache/beam)'s iceberg [extension](https://github.com/apache/beam/tree/master/sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg). Beam, and my project, are on Avro 1.11. When executing an Iceberg read, I hit the following runtime exception:

```
Exception in thread "sbt-bg-threads-1" java.lang.NoSuchMethodError: 'org.apache.avro.LogicalTypes$TimestampNanos org.apache.avro.LogicalTypes.timestampNanos()'
at org.apache.iceberg.avro.TypeToSchema.(TypeToSchema.java:50)
at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:64)
at org.apache.iceberg.avro.AvroSchemaUtil.convert(AvroSchemaUtil.java:59)
at org.apache.iceberg.GenericManifestFile.(GenericManifestFile.java:42)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at java.base/jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:77)
at java.base/jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.base/java.lang.reflect.Constructor.newInstanceWithCaller(Constructor.java:500)
at java.base/java.lang.reflect.Constructor.newInstance(Constructor.java:481)
at org.apache.iceberg.common.DynConstructors$Ctor.newInstanceChecked(DynConstructors.java:51)
at org.apache.iceberg.common.DynConstructors$Ctor.newInstance(DynConstructors.java:64)
at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:67)
at org.apache.iceberg.avro.InternalReaders$PlannedStructLikeReader.reuseOrCreate(InternalReaders.java:42)
at org.apache.iceberg.avro.ValueReaders$PlannedStructReader.read(ValueReaders.java:1025)
at org.apache.iceberg.avro.InternalReader.read(InternalReader.java:107)
at org.apache.iceberg.avro.NameMappingDatumReader.read(NameMappingDatumReader.java:57)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:263)
at org.apache.avro.file.DataFileStream.next(DataFileStream.java:248)
at org.apache.iceberg.io.CloseableIterator$1.next(CloseableIterator.java:55)
at org.apache.iceberg.relocated.com.google.common.collect.Iterators.addAll(Iterators.java:370)
at org.apache.iceberg.relocated.com.google.common.collect.Iterables.addAll(Iterables.java:332)
at org.apache.iceberg.relocated.com.google.common.collect.Lists.newLinkedList(Lists.java:261)
at org.apache.iceberg.ManifestLists.read(ManifestLists.java:42)
at org.apache.iceberg.BaseSnapshot.cacheManifests(BaseSnapshot.java:185)
at org.apache.iceberg.BaseSnapshot.dataManifests(BaseSnapshot.java:211)
at org.apache.iceberg.DataTableScan.doPlanFiles(DataTableScan.java:68)
at org.apache.iceberg.SnapshotScan.planFiles(SnapshotScan.java:139)
at org.apache.beam.sdk.io.iceberg.ScanSource.wholeTableReadTask(ScanSource.java:73)
...
```

This is happening because `timestamp-nanos` wasn't added to Avro until 1.12 (compare [1.11](https://github.com/apache/avro/blob/release-1.11.5/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L187-L197) types vs [1.12](https://github.com/apache/avro/blob/release-1.12.1/lang/java/avro/src/main/java/org/apache/avro/LogicalTypes.java#L196-L208)).

Upgrading Avro versions can be a nontrivial operation for various reasons - I was wondering if Iceberg would accept a change that gates the static initialization of `TIMESTAMP_NANO_SCHEMA` and `TIMESTAMPTZ_NANO_SCHEMA` behind an Avro version check. There's precedent for this in [parquet-java](https://github.com/apache/parquet-java/blob/apache-parquet-1.17.1/parquet-avro/src/main/java/org/apache/parquet/avro/AvroRecordConverter.java#L264-L266), however I'm not sure if Iceberg has made any commitments about Avro backwards-compatibility/if it's something that would be painful to maintain in the future.

Let me know what you think; I have a proof of concept [here](https://github.com/apache/iceberg/compare/main...clairemcginty:iceberg:avro-1.11-schema-compat?expand=1) that works. thanks!

Contributor guide

Open the contributing guide

Research direction

Start with the static initialization of TIMESTAMP_NANO_SCHEMA and TIMESTAMPTZ_NANO_SCHEMA in TypeToSchema.java, then review the AvroSchemaUtil conversion path and the linked proof of concept. Compare the compatibility approach with parquet-java's AvroRecordConverter.java precedent and verify that Iceberg reads work with Avro 1.11 while newer Avro versions retain nanosecond timestamp support.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.