apache / apache/incubator-xtable
unable to create target format Delta with source format as Iceberg when the source table is on S3
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
I followed the documentation "Creating your first interoperable table", able to build the utilities-0.1.0-SNAPSHOT-bundled.jar successfully.
**Initiated a pyspark session using below command. Spark version is 3.4.1 running on Amazon EMR 6.14**
pyspark --conf "spark.sql.extensions=org.apache.iceberg.spark.extensions.IcebergSparkSessionExtensions" --conf "spark.sql.catalog.spark_catalog=org.apache.iceberg.spark.SparkSessionCatalog" --conf "spark.sql.catalog.spark_catalog.type=hive"
**Create an Iceberg table using below commands:**
data =[("James","Smith","01012020","M","3000"),
("Michael","","02012021","M","4000"),
("Robert","Williams","03012023","M","4000"),
("Maria","Jones","04012024","F","4000"),
("Jen","Brown","05012025","F","-1")]
columns=["firstname","lastname","dob","gender","salary"]
df=spark.createDataFrame(data,columns)
spark.sql("""CREATE TABLE IF NOT EXISTS iceberg_table (firstname string,lastname string,dob string,gender string,salary string) USING iceberg""");
df.writeTo("iceberg_table").append()
**I see the data and metadata directory under the table name on s3.**
**Created my_config.yaml as mentioned in the documentation**
[my_config.txt](https://github.com/apache/incubator-xtable/files/15167270/my_config.txt)
**executed below command and see failing with metadata/version-hint.text not available**
sudo java -jar ./utilities-0.1.0-SNAPSHOT-bundled.jar --datasetConfig my_config.yaml
2024-04-30 10:24:25 INFO org.apache.xtable.conversion.ConversionController:240 - No previous InternalTable sync for target. Falling back to snapshot sync.
2024-04-30 10:24:25 WARN org.apache.iceberg.hadoop.HadoopTableOperations:325 - Error reading version hint file s3:///iceberg_table_1/metadata/version-hint.text
java.io.FileNotFoundException: No such file or directory: s3:////iceberg_table_1/metadata/version-hint.text
at org.apache.hadoop.fs.s3a.S3AFileSystem.s3GetFileStatus(S3AFileSystem.java:3801) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
at org.apache.hadoop.fs.s3a.S3AFileSystem.innerGetFileStatus(S3AFileSystem.java:3652) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
at org.apache.hadoop.fs.s3a.S3AFileSystem.extractOrFetchSimpleFileStatus(S3AFileSystem.java:5288) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
at org.apache.hadoop.fs.s3a.S3AFileSystem.lambda$executeOpen$6(S3AFileSystem.java:1578) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
at org.apache.hadoop.fs.statistics.impl.IOStatisticsBinding.lambda$trackDurationOfOperation$5(IOStatisticsBinding.java:499) ~[utilities-0.1.0-SNAPSHOT-bundled.jar:0.1.0-SNAPSHOT]
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the linked my_config.yaml and reproduce the conversion using utilities-0.1.0-SNAPSHOT-bundled.jar and the documented command. Trace the S3 Iceberg table path and the metadata/version-hint.text lookup shown in the log. Done means the Iceberg source on S3 converts to the configured Delta target without the missing metadata error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100