apache / apache/incubator-xtable

Unable to generate Delta metadata for Snowflake-created Iceberg

Open
#287 9 comments 1 reaction 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

Created Iceberg data in S3 following [this tutorial](https://medium.com/snowflake/how-to-integrate-databricks-with-snowflake-managed-iceberg-tables-7a8895c2c724)

Using Shallow Clone in Databricks with Photon enabled we can generate Delta metadata from the created Iceberg in S3. However when attempting to do the same with OneTable, we receive the following error:

```
2023-12-05 23:48:19 INFO org.apache.spark.sql.delta.InitialSnapshot:57 - [tableId=62953b04-8f92-428f-b2bf-cbaa4daf2554] Created snapshot InitialSnapshot(path=s3://brandon-kvarda-sandbox-data/iceberg/customer/_delta_log, version=-1, metadata=Metadata(30358631-745f-4dc5-b638-47570a0e2219,null,null,Format(parquet,Map()),null,List(),Map(),Some(1701820099184)), logSegment=LogSegment(s3://brandon-kvarda-sandbox-data/iceberg/customer/_delta_log,-1,List(),List(),None,-1), checksumOpt=None)
2023-12-05 23:48:19 INFO io.onetable.client.OneTableClient:236 - No previous OneTable sync for target. Falling back to snapshot sync.
2023-12-05 23:48:19 WARN org.apache.iceberg.hadoop.HadoopTableOperations:325 - Error reading version hint file s3://brandon-kvarda-sandbox-data/iceberg/customer/metadata/version-hint.text
java.lang.NumberFormatException: For input string: "1701803553322000000"
at ormatException.forInputString(NumberFormatException.java:65) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:652) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:770) ~[?:?]
at org.apache.iceberg.hadoop.HadoopTableOperations.findVersion(HadoopTableOperations.java:320) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:?]java.lang.NumberF
```

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the Snowflake-created Iceberg setup from the linked tutorial and run the OneTable sync with Photon enabled. Start with the version-hint.text read in HadoopTableOperations.findVersion and the OneTableClient snapshot-sync path, then verify that Delta metadata generation completes without the NumberFormatException.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java, spark
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.