DataFusion 1TB single machine benchmark
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Is your feature request related to a problem or challenge?
There was a challenge recently https://www.confessionsofadataguy.com/duckdb-beats-polars-for-1tb-of-data/ and I think it would be beneficial also to run the same for DataFusion
### Describe the solution you'd like
_No response_
### Describe alternatives you've considered
_No response_
### Additional context
_No response_
Contributor guide
Research direction
Start with the linked DuckDB-versus-Polars 1TB comparison and review the DataFusion repository; the issue names no file, test, or benchmark entry point. Establish a reproducible single-machine workload and report comparable DataFusion results, with the dataset, hardware, procedure, and measurements documented.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100