[MetaData Indexing via Glue ETL]
- Dominant language
- Java
- Stars
- 6.2k
- Forks
- 2.5k
- Avg merge
- 2d 8h
- Merged PRs (30d)
- 111
Description
I am using the Glue 3.0 version to execute meta dat indexing for a hudi table.
non_global_bloom_indexing = {
'hoodie.table.name': tbl_name,
'hoodie.datasource.write.precombine.field': precombine,
'hoodie.datasource.write.recordkey.field': primary_key,
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.hive_style_partitioning':'true',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.table': tbl_name,
'hoodie.datasource.hive_sync.database': db_name,
'hoodie.datasource.hive_sync.partition_fields': partition_fields,
'hoodie.datasource.hive_sync.mode': 'hms',
'hoodie.datasource.hive_sync.partition_extractor_class':'org.apache.hudi.hive.HiveStylePartitionValueExtractor',
'hoodie.datasource.write.keygenerator.class':'org.apache.hudi.keygen.SimpleKeyGenerator',
"hoodie.datasource.write.partitionpath.field": partition_fields,
'hoodie.upsert.shuffle.parallelism': 1*8*9,
'hoodie.metadata.enable' : 'true',
"hoodie.metadata.index.bloom.filter.enable":'true',
'hoodie.metadata.index.column.stats.enable' :'true',
'hoodie.metadata.index.column.stats.column.list':partition_fields,
'hoodie.enable.data.skipping':'true',
"hoodie.index.type":'BLOOM',
"hoodie.bloom.index.use.metadata" : 'true',
"hoodie.bloom.index.filter.type": "DYNAMIC_V0",
"hoodie.bloom.index.use.caching": "true",
'hoodie.datasource.write.partitionpath.urlencode': 'true'
}
As expected inside | .hoodie/ | Folder | - | - | -
below partitions are created.
| bloom_filters/ | Folder | - | - | -
| column_stats/ | Folder | - | - | -
| files/ | Folder | - | - | -
But the timeline files .indexing.requested, indexing.inflight , indexing are not created,
Contributor guide
No contributing guide indexed for this repository
Research direction
Reproduce the metadata-indexing setup with AWS Glue 3.0 and the supplied Hudi options, then inspect the .hoodie/bloom_filters, column_stats, and files folders alongside the missing indexing timeline files. Compare the observed output with the expected metadata-indexing behavior and document the cause and conditions under which the indexing files are created.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, java
- Domain
- data-engineering, databases
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100