apache / apache/incubator-xtable
AWS Glue with Iceberg Source and Delta Target
- Dominant language
- Java
- Stars
- 1.2k
- Forks
- 212
- Avg merge
- 4d 9h
- Merged PRs (30d)
- 16
Description
I seem to keep getting a `version-hint.text` error when trying to use the .jar file to sync an existing iceberg table to a delta, using AWS Glue as my catalog. Im using the `my_config.yaml` file as directed in the doc's, but i feel like im missing something.
For instance:
```
sourceFormat: ICEBERG
targetFormats:
- DELTA
datasets:
-
# (i have this set to the parent path that holds the (metadata, data and _delta_log directories), ive tried other configurations as well. The _delta_log directory gets created, but nothing gets created in it, as i keep getting the error indicated above)
tableBasePath: s3://path
tableName: my_table
```
Do i need to create a catalog.yaml and clients.yaml file as well? I noticed these but i thought they were only for implementing custom sources other than Hudi, Iceberg or Delta.
If i do need to create these files, what would i set as the `catalogImpl: ` to for a source iceberg table that is managed by aws glue?
```
catalogImpl: io.my.CatalogImpl
catalogName: name
catalogOptions: # all other options are passed through in a map
key1: value1
key2: value2
```
Thanks in advance!
Contributor guide
No contributing guide indexed for this repository
Research direction
Start with the documentation and configuration examples for my_config.yaml, catalog.yaml, and clients.yaml, then investigate the reported version-hint.text error using the AWS Glue catalog and Iceberg-to-Delta setup shown. Done means establishing whether the configuration is supported and documenting or reproducing the required catalog settings and resulting behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100