apache / apache/gluten

Lance-spark support in Gluten

Open
#12,263 25 comments 5 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

To add Lance read/write support, we have two ways:

- Use lance connector in Velox, offload the whole read/write operator to velox plan https://github.com/facebookincubator/velox/pull/16556
- Use lance-spark datasource. To read, gluten gets the arrow data from spark-spark, fwd to velox pipeline. To write, gluten gets the velox data from Velox, converts it as arrow, fwd to lance-spark. No C2R/R2C conversion in the whole process

We are going to try lance-spark as the preferred way.

### Gluten version

None

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.