aws-samples / aws-samples/aws-glue-samples

Hive table and partition basic statistics are not correctly imported into AWS Glue Catalog

Open
#31 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
1.5k
Forks
834
Avg merge
2d 17h
Merged PRs (30d)
1

Description

Hive table and partition basic statistics are not correctly imported into AWS Glue Catalog.
The statistics properties are included in the Glue table properties, however, it looks that Hive is not honoring it.

Glue migration script is capable of migrating table and partition statistics from the Hive Metastore, however, it appears that the migration script is escaping some of the characters thus making the statistics unusable in Glue catalog:
https://github.com/aws-samples/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/src/hive_metastore_migration.py#L455

----
When I created a table in Hive meta store, the table and column statistics looked like the following:

Table Parameters:
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\",\"name\":\"true\"}}
EXTERNAL TRUE
numFiles 1
numRows 2
rawDataSize 14
totalSize 16
transient_lastDdlTime 1536141689
----

----
When I ran the migration script to migrate the Hive meta store to Glue catalog, the same statistics became the following. I have found the below statistics is unusable through Hive.

Table Parameters:
COLUMN_STATS_ACCURATE \\{\\\"BASIC_STATS\\\"\\:\\\"true\\\",\\\"COLUMN_STATS\\\"\\:\\{\\\"id\\\"\\:\\\"true\\\",\\\"name\\\"\\:\\\"true\\\"\\}\\}
EXTERNAL TRUE
numFiles 1
numRows 2
rawDataSize 14
totalSize 16
transient_lastDdlTime 1536141689
----

I then manually modified table property (COLUMN_STATS_ACCURATE) in Glue console to the following and was able to convert 'COLUMN_STATS_ACCURATE' into a usable format

I didn't check the compatibility of the migrated statistics with the other EMR tools (Spark, Presto) and AWS services (Glue ETL job, Athena, Redshift Spectrum).

Regards,
Simone

Contributor guide

Open the contributing guide

Research direction

Start at utilities/Hive_metastore_migration/src/hive_metastore_migration.py around line 455 and compare the source Hive properties with the migrated Glue properties shown in the issue. Trace how COLUMN_STATS_ACCURATE is escaped, then verify that migrated table and partition statistics remain usable through the Glue Catalog.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, python
Domain
cloud, data-engineering, databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.