aws-samples / aws-samples/aws-glue-samples
Hive table and partition basic statistics are not correctly imported into AWS Glue Catalog
- Dominant language
- Python
- Stars
- 1.5k
- Forks
- 834
- Avg merge
- 2d 17h
- Merged PRs (30d)
- 1
Description
Hive table and partition basic statistics are not correctly imported into AWS Glue Catalog.
The statistics properties are included in the Glue table properties, however, it looks that Hive is not honoring it.
Glue migration script is capable of migrating table and partition statistics from the Hive Metastore, however, it appears that the migration script is escaping some of the characters thus making the statistics unusable in Glue catalog:
https://github.com/aws-samples/aws-glue-samples/blob/master/utilities/Hive_metastore_migration/src/hive_metastore_migration.py#L455
----
When I created a table in Hive meta store, the table and column statistics looked like the following:
Table Parameters:
COLUMN_STATS_ACCURATE {\"BASIC_STATS\":\"true\",\"COLUMN_STATS\":{\"id\":\"true\",\"name\":\"true\"}}
EXTERNAL TRUE
numFiles 1
numRows 2
rawDataSize 14
totalSize 16
transient_lastDdlTime 1536141689
----
----
When I ran the migration script to migrate the Hive meta store to Glue catalog, the same statistics became the following. I have found the below statistics is unusable through Hive.
Table Parameters:
COLUMN_STATS_ACCURATE \\{\\\"BASIC_STATS\\\"\\:\\\"true\\\",\\\"COLUMN_STATS\\\"\\:\\{\\\"id\\\"\\:\\\"true\\\",\\\"name\\\"\\:\\\"true\\\"\\}\\}
EXTERNAL TRUE
numFiles 1
numRows 2
rawDataSize 14
totalSize 16
transient_lastDdlTime 1536141689
----
I then manually modified table property (COLUMN_STATS_ACCURATE) in Glue console to the following and was able to convert 'COLUMN_STATS_ACCURATE' into a usable format
I didn't check the compatibility of the migrated statistics with the other EMR tools (Spark, Presto) and AWS services (Glue ETL job, Athena, Redshift Spectrum).
Regards,
Simone
Contributor guide
Research direction
Start at utilities/Hive_metastore_migration/src/hive_metastore_migration.py around line 455 and compare the source Hive properties with the migrated Glue properties shown in the issue. Trace how COLUMN_STATS_ACCURATE is escaped, then verify that migrated table and partition statistics remain usable through the Glue Catalog.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, python
- Domain
- cloud, data-engineering, databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100