Parquet read fuzzer test
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
During customer support, we noticed that Velox fails to read some parquet files, like the ones with complicated complex types or various parquet schema format. There are several issues tracked in community, including but not limited to [9242](https://github.com/facebookincubator/velox/issues/9242), [9239](https://github.com/facebookincubator/velox/issues/9239), [9238](https://github.com/facebookincubator/velox/issues/9238), [7776](https://github.com/facebookincubator/velox/issues/7776), etc. So it's quite necessary to adopt a fuzzer test against on parquet read to enhance the support of this format. We plans to:
1) port [example files](https://github.com/apache/parquet-mr/tree/master/parquet-hadoop/src/test/resources) from parquet-mr and verify it in Gluten. This is already done by PR [5345](https://github.com/apache/incubator-gluten/pull/5345) and issue [9463](https://github.com/facebookincubator/velox/issues/9463) is opened in velox upstream.
2) leverage parquet-mr [data generator](https://github.com/apache/parquet-mr/blob/master/parquet-benchmarks/src/main/java/org/apache/parquet/benchmarks/DataGenerator.java) to generate parquet files and verify scan result between Spark and Gluten.
Contributor guide
Research direction
Review the completed example-file work in PR 5345 and the parquet-mr DataGenerator referenced in the issue. Then determine how generated Parquet files should be scanned by Spark and Gluten; done means their scan results are verified against each other across the generated cases.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- java, spark
- Domain
- data-engineering, testing-qa
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100