apache / apache/gluten

[VL] useful Velox PRs not merged into upstream

Open
#11,585 18 comments 4 reactions 0 assignees View on GitHub
enhancement tracker
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

Here is a track of useful Velox PRs mostly submitted from Gluten community but not merged. It will be removed once the PR is merged automatically. Comment your PR# below if you want your PR to be tracked here. We didn't pick into gluten/velox because of the rebase effort. You may pick it on necessary

## Tracked PRs (2026-06-01)

### Bug Fixes

- [15751](https://github.com/facebookincubator/velox/pull/15751): πŸ“• [CLOSED] feat: Flush row group by buffered bytes in parquet writer
- [13734](https://github.com/facebookincubator/velox/pull/13734): πŸ“• [CLOSED] fix: Support constant value for lead/lag function
- [12684](https://github.com/facebookincubator/velox/pull/12684): πŸ“• [CLOSED] fix: Make type check reflect the corresponding logical type
- [12630](https://github.com/facebookincubator/velox/pull/12630): πŸ“• [CLOSED] fix(sparksql): Fix result mismatch cases in casting varchar to timestamp
- [12563](https://github.com/facebookincubator/velox/pull/12563): πŸ“• [CLOSED] fix: HashProbe load LazyVector before wrapping
- [10925](https://github.com/facebookincubator/velox/pull/10925): πŸ“• [CLOSED] Fix invalid 'rawResultNulls_' in SelectiveDecimalColumnReader
- [8825 ](https://github.com/facebookincubator/velox/pull/8825): πŸ“• [CLOSED] Fix Spark split function
- [8888 ](https://github.com/facebookincubator/velox/pull/8888): πŸ“• [CLOSED] Fix NaN in Spark array_intersect and array_except functions
- [15173](https://github.com/facebookincubator/velox/pull/15173): πŸ“• [CLOSED] fix(parquet): Fix reading array of row
- [8014 ](https://github.com/facebookincubator/velox/pull/8014): πŸ“• [CLOSED] Fix adapters issue caused by null config in FileHandleGenerator
- [14504](https://github.com/facebookincubator/velox/pull/14504): πŸ“• [CLOSED] fix: Fix incorrect null results when unrelated child fields are missing from the requested type in readers
- [15534](https://github.com/facebookincubator/velox/pull/15534): ⚠️ [CONFLICT] fix: Trim numeric suffix when casting string to real/double
- [15313](https://github.com/facebookincubator/velox/pull/15313): πŸ“• [CLOSED] fix(s3): Make the MinioServer instance only be initialized once
- [14277](https://github.com/facebookincubator/velox/pull/14277): πŸ“• [CLOSED] fix: An unloaded lazy vector cannot be wrapped by two different top level vectors
- [13138](https://github.com/facebookincubator/velox/pull/13138): πŸ“• [CLOSED] fix: Fix the full outer join result mismatch issue with multi duplicated match
- [11772](https://github.com/facebookincubator/velox/pull/11772): πŸ“• [CLOSED] fix: Fix the MergeSource data lost issue
- [11068](https://github.com/facebookincubator/velox/pull/11068): πŸ“• [CLOSED] Fix full outer result mismatch issue when output contains multiple matching rows
- [10402](https://github.com/facebookincubator/velox/pull/10402): πŸ“• [CLOSED] Fix the "An unsupported nested encoding was found." exception in parquet writer
- [13907](https://github.com/facebookincubator/velox/pull/13907): πŸ“• [CLOSED] feat: Fix the full outer join result mismatch issue
- [11771](https://github.com/facebookincubator/velox/pull/11771): πŸ“• [CLOSED] fix: Fix smj result mismatch issue in semi, anit and full outer join
- [15711](https://github.com/facebookincubator/velox/pull/15711): βœ… [PICKABLE] fix: Reduce memory spike in aggregate window functions
- [15343](https://github.com/facebookincubator/velox/pull/15343): πŸ“• [CLOSED] feat(parquet): Allow reading a wider integer as a narrower one
- [16164](https://github.com/facebookincubator/velox/pull/16164): βœ… [PICKABLE] fix: Def/rep level calculation for legacy Parquet lists
- [15953](https://github.com/facebookincubator/velox/pull/15953): ⚠️ [CONFLICT] fix: Use EvictingCacheMap for compiled regular expressions
- [17391](https://github.com/facebookincubator/velox/pull/17391): ⚠️ [CONFLICT] fix(parquet): Support read unknown column

### Enhancement

- [14472](https://github.com/facebookincubator/velox/pull/14472): πŸ“• [CLOSED] feat:Support multi-threaded asynchronous data upload to object storage.
- [14214](https://github.com/facebookincubator/velox/pull/14214): πŸ“• [CLOSED] feat(parquet): Support page‑level pruning
- [11285](https://github.com/facebookincubator/velox/pull/11285): πŸ“• [CLOSED] misc: Optimize the computation of sliding window kRange frame bound
- [10638](https://github.com/facebookincubator/velox/pull/10638): πŸ“• [CLOSED] Distinguish null constant and non-null constant in simple function's initialize method
- [9591 ](https://github.com/facebookincubator/velox/pull/9591): πŸ“• [CLOSED] Support offset-based timezone
- [8769 ](https://github.com/facebookincubator/velox/pull/8769): πŸ“• [CLOSED] Enable UNKNOWN type in type dispatch
- [15707](https://github.com/facebookincubator/velox/pull/15707): πŸ“• [CLOSED] feat: Enable the hash join to accept a pre-built hash table for joining
- [11808](https://github.com/facebookincubator/velox/pull/11808): πŸ“• [CLOSED] feat: Add negated hugeint filters
- [11740](https://github.com/facebookincubator/velox/pull/11740): πŸ“• [CLOSED] feat: Support decimal schema evolution in Parquet scan
- [11646](https://github.com/facebookincubator/velox/pull/11646): πŸ“• [CLOSED] feat: Support row group skip for Parquet decimal
- [5962 ](https://github.com/facebookincubator/velox/pull/5962): πŸ“• [CLOSED] feat: Support struct schema evolution matching by name
- [5464 ](https://github.com/facebookincubator/velox/pull/5464): πŸ“• [CLOSED] Add decimal type support for Spark first/last aggregate functions
- [11836](https://github.com/facebookincubator/velox/pull/11836): πŸ“• [CLOSED] feat: Optimize serializer decompress buffer for BufferInputStream
- [11824](https://github.com/facebookincubator/velox/pull/11824): πŸ“• [CLOSED] feat: Support prefix comparator in spill merge
- [11703](https://github.com/facebookincubator/velox/pull/11703): πŸ“• [CLOSED] feat(prefix sort): Eliminate null byte from prefix encoder when single sort key
- [11685](https://github.com/facebookincubator/velox/pull/11685): πŸ“• [CLOSED] feat: Support read file stream without buffer
- [11954](https://github.com/facebookincubator/velox/pull/11954): πŸ“• [CLOSED] feat: Support Spark explode outer
- [13862](https://github.com/facebookincubator/velox/pull/13862): πŸ“• [CLOSED] feat: Support spill write batch size limit
- [10456](https://github.com/facebookincubator/velox/pull/10456): πŸ“• [CLOSED] Support semi projection join type in smj
- [13041](https://github.com/facebookincubator/velox/pull/13041): ⚠️ [CONFLICT] feat: Enable the hash join to accept a pre-built hash table for joining
- [11272](https://github.com/facebookincubator/velox/pull/11272): πŸ“• [CLOSED] Support string type for PrefixSort
- [13817](https://github.com/facebookincubator/velox/pull/13817): πŸ“• [CLOSED] feat: Add zstd compression for unified compression API
- [11206](https://github.com/facebookincubator/velox/pull/11206): πŸ“• [CLOSED] Supports serializing a range of rows for UnsafeRowFast
- [7734 ](https://github.com/facebookincubator/velox/pull/7734): πŸ“• [CLOSED] Fix Parquet writer to produce evenly-sized row groups
- [15116](https://github.com/facebookincubator/velox/pull/15116): πŸ“• [CLOSED] feat: In the str_to_map Spark function, entryDelimiter and keyValueDelimiter are supported for more characters
- [15751](https://github.com/facebookincubator/velox/pull/15751): πŸ“• [CLOSED] feat: Flush row group by buffered bytes in parquet writer
- [15409](https://github.com/facebookincubator/velox/pull/15409): πŸ“• [CLOSED] feat(spilling): Fallback to timsort when allocation of prefix sort buffer memory fails during spilling
- [15290](https://github.com/facebookincubator/velox/pull/15290): πŸ“• [CLOSED] perf(spilling): Support serializing rows to avoid extracting it as vector
- [15848](https://github.com/facebookincubator/velox/pull/15848): πŸ“• [CLOSED] feat: Allow subfield rename and deletion for ORC format
- [15458](https://github.com/facebookincubator/velox/pull/15458): πŸ“• [CLOSED] perf: Optimize basic numeric upcast
- [15300](https://github.com/facebookincubator/velox/pull/15300): πŸ“• [CLOSED] feat: Add support for ORC writer
- [16514](https://github.com/facebookincubator/velox/pull/16514): βœ… [PICKABLE] feat(sparksql): Support multi-character delimiters in str_to_map
- [16547](https://github.com/facebookincubator/velox/pull/16547): βœ… [PICKABLE] perf(exec): Tiled column-major extraction for RowContainer
- [16546](https://github.com/facebookincubator/velox/pull/16546): ⚠️ [CONFLICT] perf(exec): Combine low-selectivity filter results in HashProbe
- [16545](https://github.com/facebookincubator/velox/pull/16545): ⚠️ [CONFLICT] perf(exec): Add AMAC prefetch optimization for listJoinResults
- [16852](https://github.com/facebookincubator/velox/pull/16852): βœ… [PICKABLE] perf(operator): Batch merge spilled rows in HashAggregation

### Iceberg

- [14276](https://github.com/facebookincubator/velox/pull/14276): πŸ“• [CLOSED] feat(iceberg): Add Iceberg all functions

### Json

- [11433](https://github.com/facebookincubator/velox/pull/11433): πŸ“• [CLOSED] Fix JSON parser to allow control characters in JSON string input
- [12892](https://github.com/facebookincubator/velox/pull/12892): πŸ“• [CLOSED] feat(sparksql): Support wildcard in json path for get_json_object function
- [5179 ](https://github.com/facebookincubator/velox/pull/5179): πŸ“• [CLOSED] Optimize get_json_object Spark function using simdjson
- [6016 ](https://github.com/facebookincubator/velox/pull/6016): πŸ“• [CLOSED] Reject duplicated keys in abstract join node
- [14801](https://github.com/facebookincubator/velox/pull/14801): πŸ“• [CLOSED] fix: Minify JSON objects/arrays in Spark get_json_object

### Regexp

- [10279](https://github.com/facebookincubator/velox/pull/10279): πŸ“• [CLOSED] Introduce Hyperscan lib to implement regexp functions
- [8387 ](https://github.com/facebookincubator/velox/pull/8387): πŸ“• [CLOSED] Fix signature of regexp_replace Spark function and register it in Spark function registry

### Spark Functions

- [7555 ](https://github.com/facebookincubator/velox/pull/7555): πŸ“• [CLOSED] Add date_format Spark function
- [6296 ](https://github.com/facebookincubator/velox/pull/6296): πŸ“• [CLOSED] Add SparkSQL url_decode function
- [9719 ](https://github.com/facebookincubator/velox/pull/9719): πŸ“• [CLOSED] Support `allowPrecisionLoss` in Spark decimal ops
- [11304](https://github.com/facebookincubator/velox/pull/11304): πŸ“• [CLOSED] Register re-usable Presto date_trunc functions for Spark
- [9714 ](https://github.com/facebookincubator/velox/pull/9714): πŸ“• [CLOSED] Add session timezone getter
- [7086 ](https://github.com/facebookincubator/velox/pull/7086): πŸ“• [CLOSED] Support Spark array_union function
- [7083 ](https://github.com/facebookincubator/velox/pull/7083): πŸ“• [CLOSED] Add Spark quarter function
- [12780](https://github.com/facebookincubator/velox/pull/12780): πŸ“• [CLOSED] feat: Add Spark to_pretty_string function
- [12763](https://github.com/facebookincubator/velox/pull/12763): πŸ“• [CLOSED] feat: Add Spark make_dt_interval function
- [12762](https://github.com/facebookincubator/velox/pull/12762): πŸ“• [CLOSED] feat: Add CAST(interval year month as integer)
- [12521](https://github.com/facebookincubator/velox/pull/12521): πŸ“• [CLOSED] [Velox] Add Support for Day Time Interval Type
- [12369](https://github.com/facebookincubator/velox/pull/12369): πŸ“• [CLOSED] feat: Add support for Timestamp to Integral for Spark
- [12230](https://github.com/facebookincubator/velox/pull/12230): πŸ“• [CLOSED] feat: Add support for double to timestamp cast for Spark
- [12229](https://github.com/facebookincubator/velox/pull/12229): πŸ“• [CLOSED] feat: Add Spark support to cast double to timestamp
- [10788](https://github.com/facebookincubator/velox/pull/10788): πŸ“• [CLOSED] Add Spark split_part function
- [8692 ](https://github.com/facebookincubator/velox/pull/8692): πŸ“• [CLOSED] Add map_from_entries Spark function
- [10359](https://github.com/facebookincubator/velox/pull/10359): πŸ“• [CLOSED] Add SparkSql function to_pretty_string
- [12749](https://github.com/facebookincubator/velox/pull/12749): πŸ“• [CLOSED] feat: Register spark map_from_entries function
- [11033](https://github.com/facebookincubator/velox/pull/11033): πŸ“• [CLOSED] Support sparksql approx_percentile
- [10280](https://github.com/facebookincubator/velox/pull/10280): πŸ“• [CLOSED] feat: Register function for map_from_arrays for SparkSQL
- [11114](https://github.com/facebookincubator/velox/pull/11114): πŸ“• [CLOSED] Support all patterns for Spark CAST(varchar as timestamp)
- [12512](https://github.com/facebookincubator/velox/pull/12512): πŸ“• [CLOSED] fix(expr): Align cast from decimal to float/double with Spark and Presto
- [4859 ](https://github.com/facebookincubator/velox/pull/4859): πŸ“• [CLOSED] Add months_between Spark function
- [4830 ](https://github.com/facebookincubator/velox/pull/4830): πŸ“• [CLOSED] Add next_day Spark function
- [10641](https://github.com/facebookincubator/velox/pull/10641): πŸ“• [CLOSED] Support overflow in Timestamp::toTimeZone method
- [5419 ](https://github.com/facebookincubator/velox/pull/5419): πŸ“• [CLOSED] Add substring_index spark function
- [11126](https://github.com/facebookincubator/velox/pull/11126): πŸ“• [CLOSED] Skip overflow check for decimal add in agg function
- [9272 ](https://github.com/facebookincubator/velox/pull/9272): πŸ“• [CLOSED] Add normalize_nan Spark function
- [8356 ](https://github.com/facebookincubator/velox/pull/8356): πŸ“• [CLOSED] Add nanvl Spark function
- [7204 ](https://github.com/facebookincubator/velox/pull/7204): πŸ“• [CLOSED] Add corr Spark function

Contributor guide

Open the contributing guide

Research direction

No project files or tests are named; the issue is a tracking list of Velox pull requests. Start by reviewing entries marked PICKABLE or CONFLICT and checking their current upstream status. Done means the list accurately reflects actionable, unmerged work and removes or updates entries as their pull requests change.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, spark
Domain
backend, data-engineering
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.