apache / apache/incubator-xtable

AWS Glue with Iceberg Source and Delta Target

Open
#354 41 comments 0 reactions 0 assignees View on GitHub
Dominant language
Java
Stars
1.2k
Forks
212
Avg merge
4d 9h
Merged PRs (30d)
16

Description

I seem to keep getting a `version-hint.text` error when trying to use the .jar file to sync an existing iceberg table to a delta, using AWS Glue as my catalog. Im using the `my_config.yaml` file as directed in the doc's, but i feel like im missing something.

For instance:
```
sourceFormat: ICEBERG
targetFormats:
- DELTA
datasets:
-
# (i have this set to the parent path that holds the (metadata, data and _delta_log directories), ive tried other configurations as well. The _delta_log directory gets created, but nothing gets created in it, as i keep getting the error indicated above)
tableBasePath: s3://path
tableName: my_table
```

Do i need to create a catalog.yaml and clients.yaml file as well? I noticed these but i thought they were only for implementing custom sources other than Hudi, Iceberg or Delta.

If i do need to create these files, what would i set as the `catalogImpl: ` to for a source iceberg table that is managed by aws glue?
```
catalogImpl: io.my.CatalogImpl
catalogName: name
catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```

Thanks in advance!

Contributor guide

No contributing guide indexed for this repository

Research direction

Start with the documentation and configuration examples for my_config.yaml, catalog.yaml, and clients.yaml, then investigate the reported version-hint.text error using the AWS Glue catalog and Iceberg-to-Delta setup shown. Done means establishing whether the configuration is supported and documenting or reproducing the required catalog settings and resulting behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.