Lance-spark support in Gluten
Open
enhancement
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
To add Lance read/write support, we have two ways:
- Use lance connector in Velox, offload the whole read/write operator to velox plan https://github.com/facebookincubator/velox/pull/16556
- Use lance-spark datasource. To read, gluten gets the arrow data from spark-spark, fwd to velox pipeline. To write, gluten gets the velox data from Velox, converts it as arrow, fwd to lance-spark. No C2R/R2C conversion in the whole process
We are going to try lance-spark as the preferred way.
### Gluten version
None
Contributor guide
Assessment
This issue has not been assessed yet.