apache / apache/incubator-graphar

Fully utilize the features of different file formats for improved efficiency

Open
#75 0 comments 0 reactions 0 assignees View on GitHub
improvement
Dominant language
C++
Stars
373
Forks
93
Avg merge
7d 21h
Merged PRs (30d)
7

Description

**Is your feature request related to a problem? Please describe.**
GraphAr supports the file formats of CSV, ORC and Parquet currently, and it's going to support more file types such as json, hdf5 and avro. For enhancing the efficiency of reading/writing and storing of the data, the features of different file formats should be considered and fully utilized, for example, applying the most appropriate compression and encoding scheme to the data, or enable filter pushdown to improve query performance.

**Describe the solution you'd like**
A clear and concise description of what you want to happen.

**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.

**Additional context**
Add any other context or screenshots about the feature request here.

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. First clarify which supported formats and efficiency goals are in scope, then inspect the CSV, ORC, and Parquet read/write paths and establish benchmarks; done means format-appropriate compression or encoding and filter pushdown are implemented and measured.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering, performance
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.