[VL] Port AttachDistributedSequenceExec to Velox backend
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 85
Description
### Description
Spark's `AttachDistributedSequenceExec` prepends a contiguous, globally increasing `Long` id column to its child output. It is used by pandas-on-Spark's `distributed-sequence` default index and by `DataFrame.zipWithIndex`. Today Gluten falls back to vanilla Spark for this operator, which forces a columnar → row transition that dominates runtime for wide / nested-typed inputs.
This issue tracks porting the operator to the Velox backend so that it runs end-to-end on columnar batches.
Contributor guide
Research direction
Start by tracing Spark's AttachDistributedSequenceExec and how the Velox backend handles this operator. Verify the operator's contiguous, globally increasing Long id column behavior, then confirm it runs end-to-end on columnar batches without falling back to vanilla Spark.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala
- Domain
- backend, data-engineering, distributed-systems
- Issue type
- Feature
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100