[VL] allow to read from CSV/Hive TEXT natively
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
CSV/TEXT format is a still commonly used some workloads(legacy workloads like HIVE job). Velox does not support CSV/TEXT yet and it may require big efforts on support this. Instead we could support this via arrow dataset(https://github.com/oap-project/gazelle_plugin/tree/main/arrow-data-source) and export the arrow columnar vector as Velox columnar vector after scan.
Contributor guide
Research direction
Start by reviewing the arrow-data-source implementation linked in the issue and how its scan output could connect to Velox vectors. Define the CSV/TEXT and Hive TEXT scope, then verify that scans can read these formats through Arrow and export usable Velox columnar vectors.
Written by the indexing model from the issue text.
Assessment
- Domain
- data-engineering
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100