apache / apache/hudi

[MetaData Indexing via Glue ETL]

Open
#8,954 7 comments 0 reactions 0 assignees View on GitHub
area:aws area:index priority:high
Dominant language
Java
Stars
6.2k
Forks
2.5k
Avg merge
2d 8h
Merged PRs (30d)
111

Description

I am using the Glue 3.0 version to execute meta dat indexing for a hudi table.

non_global_bloom_indexing = {
'hoodie.table.name': tbl_name,
'hoodie.datasource.write.precombine.field': precombine,
'hoodie.datasource.write.recordkey.field': primary_key,
'hoodie.datasource.write.table.type': 'COPY_ON_WRITE',
'hoodie.datasource.write.hive_style_partitioning':'true',
'hoodie.datasource.write.operation': 'upsert',
'hoodie.datasource.hive_sync.enable': 'true',
'hoodie.datasource.hive_sync.table': tbl_name,
'hoodie.datasource.hive_sync.database': db_name,
'hoodie.datasource.hive_sync.partition_fields': partition_fields,
'hoodie.datasource.hive_sync.mode': 'hms',
'hoodie.datasource.hive_sync.partition_extractor_class':'org.apache.hudi.hive.HiveStylePartitionValueExtractor',
'hoodie.datasource.write.keygenerator.class':'org.apache.hudi.keygen.SimpleKeyGenerator',
"hoodie.datasource.write.partitionpath.field": partition_fields,
'hoodie.upsert.shuffle.parallelism': 1*8*9,
'hoodie.metadata.enable' : 'true',
"hoodie.metadata.index.bloom.filter.enable":'true',
'hoodie.metadata.index.column.stats.enable' :'true',
'hoodie.metadata.index.column.stats.column.list':partition_fields,
'hoodie.enable.data.skipping':'true',
"hoodie.index.type":'BLOOM',
"hoodie.bloom.index.use.metadata" : 'true',
"hoodie.bloom.index.filter.type": "DYNAMIC_V0",
"hoodie.bloom.index.use.caching": "true",

'hoodie.datasource.write.partitionpath.urlencode': 'true'
}

As expected inside  | .hoodie/ | Folder | - | - | -
below partitions are created.
  | bloom_filters/ | Folder | - | - | -
  | column_stats/ | Folder | - | - | -
  | files/ | Folder | - | - | -

But the timeline files .indexing.requested, indexing.inflight , indexing are not created,

Contributor guide

No contributing guide indexed for this repository

Research direction

Reproduce the metadata-indexing setup with AWS Glue 3.0 and the supplied Hudi options, then inspect the .hoodie/bloom_filters, column_stats, and files folders alongside the missing indexing timeline files. Compare the observed output with the expected metadata-indexing behavior and document the cause and conditions under which the indexing files are created.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, java
Domain
data-engineering, databases
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.