apache / apache/incubator-xtable
Unable to generate Delta metadata for Snowflake-created Iceberg
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
Created Iceberg data in S3 following [this tutorial](https://medium.com/snowflake/how-to-integrate-databricks-with-snowflake-managed-iceberg-tables-7a8895c2c724)
Using Shallow Clone in Databricks with Photon enabled we can generate Delta metadata from the created Iceberg in S3. However when attempting to do the same with OneTable, we receive the following error:
```
2023-12-05 23:48:19 INFO org.apache.spark.sql.delta.InitialSnapshot:57 - [tableId=62953b04-8f92-428f-b2bf-cbaa4daf2554] Created snapshot InitialSnapshot(path=s3://brandon-kvarda-sandbox-data/iceberg/customer/_delta_log, version=-1, metadata=Metadata(30358631-745f-4dc5-b638-47570a0e2219,null,null,Format(parquet,Map()),null,List(),Map(),Some(1701820099184)), logSegment=LogSegment(s3://brandon-kvarda-sandbox-data/iceberg/customer/_delta_log,-1,List(),List(),None,-1), checksumOpt=None)
2023-12-05 23:48:19 INFO io.onetable.client.OneTableClient:236 - No previous OneTable sync for target. Falling back to snapshot sync.
2023-12-05 23:48:19 WARN org.apache.iceberg.hadoop.HadoopTableOperations:325 - Error reading version hint file s3://brandon-kvarda-sandbox-data/iceberg/customer/metadata/version-hint.text
java.lang.NumberFormatException: For input string: "1701803553322000000"
at ormatException.forInputString(NumberFormatException.java:65) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:652) ~[?:?]
at java.lang.Integer.parseInt(Integer.java:770) ~[?:?]
at org.apache.iceberg.hadoop.HadoopTableOperations.findVersion(HadoopTableOperations.java:320) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:?]java.lang.NumberF
```
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the Snowflake-created Iceberg setup from the linked tutorial and run the OneTable sync with Photon enabled. Start with the version-hint.text read in HadoopTableOperations.findVersion and the OneTableClient snapshot-sync path, then verify that Delta metadata generation completes without the NumberFormatException.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java, spark
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100