Blog post about parquet vs custom file formats / test Optimized Parquet with ClickBench
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
https://x.com/andrewlamb1111/status/1925537738360504663
> ClickBench keeps me convinced that Parquet can be quite fast. There is only a 2.3x performance difference vs [@duckdb](https://x.com/duckdb) 's own format and unoptimized parquet: [https://tinyurl.com/5aexvsfw](https://t.co/NOXq3AAFlk). I am surprised that the (closed source) Umbra only reports 3.3x faster than DuckDB on parquet
### Describe the solution you'd like
I would love to make a blog post about how much faster/slower custom file formats are compared to parquet. I am typing this ticket now that it is on my mind so I don't forget it.
The basic thesis is that
* Custom file formats only get you XX% more performance than parquet
* Many of the historic performance differences are due to engineering investment rather than format
* Parquet has many other benefits (like a very large ecosystem)
==> therefore parquet is the format that really matters
### Describe alternatives you've considered
The core of the post would be to compare
1. A propretary format (like duckdb/umbra)
2. normal parquet
3. "optimized parquet"
I think we could basically use the https://github.com/ClickHouse/ClickBench dataset and queries (and results from the proprietary systems)
The thing that is needed is to generate "optimized parquet" numbers.
The [partitioned parquet files](https://github.com/ClickHouse/ClickBench?tab=readme-ov-file#data-loading) from ClickBench are not optimized. Specifically they:
1. Are not sorted in any way
2. Do not have a page index (Offset index)
3. Use snappy compression
A fun experiment might be to "fix" the clickbench partitioned dataset by
1. resorting and writing with page indexes (could use a bunch of DataFusion `COPY` commands pretty easily to do this). The sort order should be some subset of the predicate columns. Perhaps EventTime and then maybe SearchPhrase / URL.
2. disabling compression
### Additional context
_No response_
Contributor guide
Research direction
Start with the ClickBench dataset and queries, the partitioned Parquet files, and the proposed DataFusion COPY commands. Compare proprietary format results with normal and optimized Parquet, testing sorting, page indexes, and compression choices. Done means producing optimized Parquet numbers and a blog post explaining the performance and ecosystem tradeoffs.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, documentation, performance
- Issue type
- Documentation
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100