apache / apache/incubator-xtable

NullPointerException when using Snowflake Catalog as source and Delta Lake as target

Open
#641 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

I've followed the steps within this [blog](https://medium.com/@sagarlakshmipathy/how-to-interoperate-with-snowflake-and-open-table-formats-e70f453787a2) to generate Delta Lake metadata from a Snowflake managed Iceberg table but run into the following NullPointerException:
java.lang.NullPointerException: null
at org.apache.xtable.iceberg.IcebergColumnStatsConverter.lambda$fromIceberg$2(IcebergColumnStatsConverter.java:106) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.apache.xtable.iceberg.IcebergColumnStatsConverter.fromIceberg(IcebergColumnStatsConverter.java:119) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergDataFileExtractor.fromIceberg(IcebergDataFileExtractor.java:64) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergDataFileExtractor.fromIceberg(IcebergDataFileExtractor.java:53) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergConversionSource.fromIceberg(IcebergConversionSource.java:175) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:156) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:282) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.syncTableFormats(ConversionController.java:202) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:105) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.utilities.RunSync.main(RunSync.java:191) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]

### Are you willing to submit PR?

- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Snowflake-managed Iceberg to Delta Lake conversion described in the linked blog, then inspect IcebergColumnStatsConverter.fromIceberg at line 106 and its callers in IcebergDataFileExtractor and IcebergConversionSource. Done means the conversion completes without the reported NullPointerException and generates the Delta Lake metadata.

Written by the indexing model from the issue text.

Assessment

Tech stack
java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.