apache / apache/incubator-xtable

Snowflake on Azure Open Catalog does not work

Open
#667 0 comments 0 reactions 0 assignees View on GitHub
bug
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

### Search before asking

- [x] I had searched in the [issues](https://github.com/apache/incubator-xtable/issues?q=is%3Aissue) and found no similar issues.

### Please describe the bug 🐞

I ran into an issue while using Snowflake's (Azure) polaris catalog. Documenting here.

```
java -jar xtable-utilities/target/xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar --datasetConfig ./demo/config.yml -i ./demo/catalog.yaml
```

## Error

```
2025-03-17 16:41:17 INFO org.apache.iceberg.metrics.LoggingMetricsReporter:38 - Received metrics report: ScanReport{tableName=azure_ext.ICEBIRG_DEMO.PUBLIC.TEST_TABLE_MANAGED_ST, snapshotId=8300558301514482769, filter=true, schemaId=0, projectedFieldIds=[1], projectedFieldNames=[COL1], scanMetrics=ScanMetricsResult{totalPlanningDuration=TimerResult{timeUnit=NANOSECONDS, totalDuration=PT0.547000254S, count=1}, resultDataFiles=CounterResult{unit=COUNT, value=1}, resultDeleteFiles=CounterResult{unit=COUNT, value=0}, totalDataManifests=CounterResult{unit=COUNT, value=1}, totalDeleteManifests=CounterResult{unit=COUNT, value=0}, scannedDataManifests=CounterResult{unit=COUNT, value=1}, skippedDataManifests=CounterResult{unit=COUNT, value=0}, totalFileSizeInBytes=CounterResult{unit=BYTES, value=512}, totalDeleteFileSizeInBytes=CounterResult{unit=BYTES, value=0}, skippedDataFiles=CounterResult{unit=COUNT, value=0}, skippedDeleteFiles=CounterResult{unit=COUNT, value=0}, scannedDeleteManifests=CounterResult{unit=COUNT, value=0}, skippedDeleteManifests=CounterResult{unit=COUNT, value=0}, indexedDeleteFiles=CounterResult{unit=COUNT, value=0}, equalityDeleteFiles=CounterResult{unit=COUNT, value=0}, positionalDeleteFiles=CounterResult{unit=COUNT, value=0}}, metadata={iceberg-version=Apache Iceberg 1.4.2 (commit f6bb9173b13424d77e7ad8439b5ef9627e530cb2)}}
2025-03-17 16:41:17 ERROR org.apache.xtable.utilities.RunSync:239 - Error running sync for abfss:///my-container@myaccount.dfs.core.windows.net/hub/public/test_table_managed_st
java.lang.NullPointerException: null
at org.apache.xtable.iceberg.IcebergColumnStatsConverter.lambda$fromIceberg$2(IcebergColumnStatsConverter.java:106) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) ~[?:?]
at java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177) ~[?:?]
at java.util.ArrayList$ArrayListSpliterator.forEachRemaining(ArrayList.java:1655) ~[?:?]
at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) ~[?:?]
at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) ~[?:?]
at java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) ~[?:?]
at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) ~[?:?]
at java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) ~[?:?]
at org.apache.xtable.iceberg.IcebergColumnStatsConverter.fromIceberg(IcebergColumnStatsConverter.java:119) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergDataFileExtractor.fromIceberg(IcebergDataFileExtractor.java:64) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergDataFileExtractor.fromIceberg(IcebergDataFileExtractor.java:53) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergConversionSource.fromIceberg(IcebergConversionSource.java:176) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.iceberg.IcebergConversionSource.getCurrentSnapshot(IcebergConversionSource.java:156) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.spi.extractor.ExtractFromSource.extractSnapshot(ExtractFromSource.java:38) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.syncSnapshot(ConversionController.java:282) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.syncTableFormats(ConversionController.java:202) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.conversion.ConversionController.sync(ConversionController.java:105) ~[xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.utilities.RunSync.runSync(RunSync.java:237) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
at org.apache.xtable.utilities.RunSync.main(RunSync.java:160) [xtable-utilities_2.12-0.2.0-SNAPSHOT-bundled.jar:0.2.0-SNAPSHOT]
```

## config.yaml

```sourceFormat: ICEBERG
targetFormats:
- DELTA
datasets:
-
tableBasePath: abfss://my-container@myaccount.dfs.core.windows.net/hub/public/test_table_managed_st
tableDataPath: abfss://my-container@myaccount.dfs.core.windows.net/hub/public/test_table_managed_st/data # replace this with gs://path/to/delta-dataset/people if your data is in GCS.
tableName: TEST_TABLE_MANAGED_ST
namespace: ICEBIRG_DEMO.PUBLIC
```

## catalog.yaml

```
catalogImpl: org.apache.iceberg.rest.RESTCatalog
catalogName: azure_ext
catalogOptions:

warehouse: azure_ext
uri: https://.snowflakecomputing.com/polaris/api/catalog
credential: <>:<>
header.X-Iceberg-Access-Delegation: vended-credentials
scope: PRINCIPAL_ROLE:ALL
client.region: japan-east.azure
```

​This error occurs with public access to the blob enabled, so I don't believe it's related to credential vending.

### Are you willing to submit PR?

- [ ] I am willing to submit a PR!
- [ ] I am willing to submit a PR but need help getting started!

### Code of Conduct

- [x] I agree to follow this project's [Code of Conduct](https://www.apache.org/foundation/policies/conduct)

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by reproducing the supplied java -jar command with the shown config.yaml and catalog.yaml. Then inspect org.apache.xtable.iceberg.IcebergColumnStatsConverter.fromIceberg, especially line 106, followed by IcebergDataFileExtractor.fromIceberg. Done means syncing the Snowflake Polaris table on Azure completes without the NullPointerException.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, java
Domain
cloud, data, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
42/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.