apache / apache/gluten

[VL] allow to read from CSV/Hive TEXT natively

Open
#5,414 0 comments 2 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

CSV/TEXT format is a still commonly used some workloads(legacy workloads like HIVE job). Velox does not support CSV/TEXT yet and it may require big efforts on support this. Instead we could support this via arrow dataset(https://github.com/oap-project/gazelle_plugin/tree/main/arrow-data-source) and export the arrow columnar vector as Velox columnar vector after scan.

Contributor guide

Open the contributing guide

Research direction

Start by reviewing the arrow-data-source implementation linked in the issue and how its scan output could connect to Velox vectors. Define the CSV/TEXT and Hive TEXT scope, then verify that scans can read these formats through Arrow and export usable Velox columnar vectors.

Written by the indexing model from the issue text.

Assessment

Domain
data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.