aws-samples / aws-samples/aws-glue-samples
Issue with direct migration from hive metastore to glue catalog
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 834
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 1
Description
While migrating table with OpenCSVSerde from hive metastore to glue catalog it brings extra character on serde properties. Like hive metastore has * however glue catalog will have \*.
I think the problem is creted by the below function (hive_metastore_migration.py)
def udf_escape_chars(param_value):
ret_param_value = param_value.replace('\\', '\\\\')\
.replace('|', '\\|')\
.replace('"', '\\"')\
.replace('{', '\\{')\
.replace(':', '\\:')\
.replace('}', '\\}')
Contributor guide
Research direction
Start in hive_metastore_migration.py at udf_escape_chars and trace where it is used during Hive metastore to Glue Catalog migration. Reproduce the OpenCSVSerde case and compare the serde properties before and after migration; done means characters such as * are not given an extra escape in the Glue Catalog values.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 2/5
- Estimated time
- 1-3 hours
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 38/100