apache / apache/gluten

Parquet read fuzzer test

Open
#5,434 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

During customer support, we noticed that Velox fails to read some parquet files, like the ones with complicated complex types or various parquet schema format. There are several issues tracked in community, including but not limited to [9242](https://github.com/facebookincubator/velox/issues/9242), [9239](https://github.com/facebookincubator/velox/issues/9239), [9238](https://github.com/facebookincubator/velox/issues/9238), [7776](https://github.com/facebookincubator/velox/issues/7776), etc. So it's quite necessary to adopt a fuzzer test against on parquet read to enhance the support of this format. We plans to:
1) port [example files](https://github.com/apache/parquet-mr/tree/master/parquet-hadoop/src/test/resources) from parquet-mr and verify it in Gluten. This is already done by PR [5345](https://github.com/apache/incubator-gluten/pull/5345) and issue [9463](https://github.com/facebookincubator/velox/issues/9463) is opened in velox upstream.
2) leverage parquet-mr [data generator](https://github.com/apache/parquet-mr/blob/master/parquet-benchmarks/src/main/java/org/apache/parquet/benchmarks/DataGenerator.java) to generate parquet files and verify scan result between Spark and Gluten.

Contributor guide

Open the contributing guide

Research direction

Review the completed example-file work in PR 5345 and the parquet-mr DataGenerator referenced in the issue. Then determine how generated Parquet files should be scanned by Spark and Gluten; done means their scan results are verified against each other across the generated cases.

Written by the indexing model from the issue text.

Assessment

Tech stack
java, spark
Domain
data-engineering, testing-qa
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.