apache / apache/gluten

Parquet metadata check limit optimization

Open
#11,782 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Now the validation metadata config(spark.gluten.sql.fallbackUnexpectedMetadataParquet) is default false, if set to true, for each root path, we check the file limit (spark.gluten.sql.fallbackUnexpectedMetadataParquet.limit), if the number of partitions are too much, the validation will be expensive.

The possible solution is to sample the rootPaths to select some files.

The sample file limit should be decided by file total limit number and the total file number in root paths, the latter should be decided by the percentage.

### Gluten version

None

Contributor guide

Open the contributing guide

Research direction

Start by tracing the validation controlled by spark.gluten.sql.fallbackUnexpectedMetadataParquet and its limit setting. Determine how root paths, partition counts, and file totals are currently evaluated; done should mean sampling avoids expensive validation while respecting both the configured total file limit and the requested percentage.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
data
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.