apache / apache/gluten

[VL] Port AttachDistributedSequenceExec to Velox backend

Open
#12,187 0 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 21h
Merged PRs (30d)
85

Description

### Description

Spark's `AttachDistributedSequenceExec` prepends a contiguous, globally increasing `Long` id column to its child output. It is used by pandas-on-Spark's `distributed-sequence` default index and by `DataFrame.zipWithIndex`. Today Gluten falls back to vanilla Spark for this operator, which forces a columnar → row transition that dominates runtime for wide / nested-typed inputs.

This issue tracks porting the operator to the Velox backend so that it runs end-to-end on columnar batches.

Contributor guide

Open the contributing guide

Research direction

Start by tracing Spark's AttachDistributedSequenceExec and how the Velox backend handles this operator. Verify the operator's contiguous, globally increasing Long id column behavior, then confirm it runs end-to-end on columnar batches without falling back to vanilla Spark.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala
Domain
backend, data-engineering, distributed-systems
Issue type
Feature
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.