apache / apache/incubator-graphar
Fully utilize the features of different file formats for improved efficiency
- Dominant language
- C++
- Stars
- 373
- Forks
- 93
- Avg merge
- 7d 21h
- Merged PRs (30d)
- 7
Description
**Is your feature request related to a problem? Please describe.**
GraphAr supports the file formats of CSV, ORC and Parquet currently, and it's going to support more file types such as json, hdf5 and avro. For enhancing the efficiency of reading/writing and storing of the data, the features of different file formats should be considered and fully utilized, for example, applying the most appropriate compression and encoding scheme to the data, or enable filter pushdown to improve query performance.
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
Contributor guide
Research direction
No files, tests, or entry points are named. First clarify which supported formats and efficiency goals are in scope, then inspect the CSV, ORC, and Parquet read/write paths and establish benchmarks; done means format-appropriate compression or encoding and filter pushdown are implemented and measured.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering, performance
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100