Project idea: How good is well-configured parquet compared to proprietary file formats?

I had a nice talk with @alamb today, and he suggested a fun project

Observations: proprietary file formats usually perform faster than Parquet, as shown in ClickBench

There're two hypothesis to explain this performance gap:

1. Those formats somehow do better than Parquet
2. Those systems load the raw ClickBench's Parquet file and make sorting/clustering when convert them into proprietary formats. Therefore, a lot of the benefits come from sorting/clustering.

The goal of this project is to:

1. Quantify how much benefits come from the format itself versus the pre-processing of the file
2. (extend goal) figure out exactly which features of the proprietary file format are better than Parquet.

Concrete steps:

• Run DataFusion with unmodified ClickBench
• Make a sorted/clustered ClickBench Parquet file
• Run DataFusion with this new Parquet file, and compare the performance
• Profile and analyze where did the time go

Fun challenge: how much performance gain you can get by simply tuning the Parquet configurations -- page index, bloom filter, clustering, compression etc.

More contexts: https://x.com/andrewlamb1111/status/1925537738360504663

No source files or tests are named. Start with the ClickBench dataset and the unmodified DataFusion run, then compare it with sorted or clustered Parquet and tuned Parquet configurations. Done means quantifying format versus preprocessing gains, profiling the costs, and documenting which Parquet settings or proprietary-format features explain the difference.