aws-samples / aws-samples/aws-glue-samples

Issue with direct migration from hive metastore to glue catalog

Open
#52 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
834
Avg merge
2d 17h
Merged PRs (30d)
1

Description

While migrating table with OpenCSVSerde from hive metastore to glue catalog it brings extra character on serde properties. Like hive metastore has * however glue catalog will have \*.

I think the problem is creted by the below function (hive_metastore_migration.py)

def udf_escape_chars(param_value):
ret_param_value = param_value.replace('\\', '\\\\')\
.replace('|', '\\|')\
.replace('"', '\\"')\
.replace('{', '\\{')\
.replace(':', '\\:')\
.replace('}', '\\}')

Contributor guide

Open the contributing guide

Research direction

Start in hive_metastore_migration.py at udf_escape_chars and trace where it is used during Hive metastore to Glue Catalog migration. Reproduce the OpenCSVSerde case and compare the serde properties before and after migration; done means characters such as * are not given an extra escape in the Glue Catalog values.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
2/5
Estimated time
1-3 hours
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.