apache / apache/gluten

[VL] Unsupported spark function list [please leave a comment if you plan to pick some]

Open
#4,039 93 comments 0 reactions 0 assignees View on GitHub
enhancement
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
2d 14h
Merged PRs (30d)
80

Description

### Description

Here listed spark functions still not supported by Gluten Velox backend. **Please leave a comment if you'd like to pick some.** In the below list, [√] means someone is working in progress for the corresponding function.
You can find all functions' support status from [this gluten doc](https://github.com/oap-project/gluten/blob/main/docs/velox-backend-support-progress.md#function-support).

To avoid duplicate work, before starting, please check whether a PR has been submitted in Velox community or whether it has already been implemented in Velox who holds most sql functions in its sparksql folder & prestosql folder.
### Reference:
* [spark sql expression ](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-functions/sql-expression-schema.md)
* [spark built-in functions](https://spark.apache.org/docs/latest/api/sql/index.html)
----------
- [x] percentile_approx/approx_percentile (WIP, guangxin)
- [x] concat_ws (PR ready, https://github.com/facebookincubator/velox/pull/8854)
- [x] unix_timestamp: "Only supports string type, with session timezone considered, todo: support date type"
- [x] locate
- [x] parse_url (PR drafted, not merged)
- [x] urldecoder: "UDF, supported by spark as a built-in function since 3.4.0."
- [ ] normalizenanandzero
- [x] arrayintersects
- [ ] default.json_split (udf, no need to impl.): "external UDF"
- [ ] parsejsonarray: "external UDF"
- [x] struct
- [x] percentile (@Yohahaha)
- [x] first/first_value (@JkSelf)
- [x] last/last_value (@JkSelf)
- [x] posexplode (WIP, @marin-ma)
- [x] trunc (WIP, HannanKan)
- [x] months_between (PR ready)
- [ ] stack
- [x] grouping_id
- [x] printf (@Surbhi-Vijay)
- [x] space (WIP, rhh777)
- [x] inline (WIP, @marin-ma)
- [x] to_unix_timestamp: "Only supports string type, with session timezone considered. todo: support date type"
- [x] from_csv (@mudit-97)
- [x] from_json (https://github.com/facebookincubator/velox/pull/11709)
- [x] to_json (@wecharyu)
- [x] json_object_keys (@dcoliversun)
- [ ] json_tuple
- [x] schema_of_csv
- [x] schema_of_json
- [x] to_csv
- [x] make_ym_interval (WIP, @marin-ma)
- [x] make_timestamp (WIP, @marin-ma)
- [x] make_interval
- [x] make_dt_interval
- [x] from_utc_timestamp (@acvictor)
- [ ] extract
- [ ] exists (@lyy-pineapple)
- [ ] date_part
- [ ] zip_with
- [x] transform (@Yohahaha)
- [ ] transform_keys
- [ ] transform_values
- [x] map_from_entries (WIP, MaYan)
- [x] map_filter (WIP, MaYan)
- [x] map_entries (Done, by MaYan)
- [x] forall (@lyy-pineapple)
- [x] flatten (@ivoson)
- [ ] filter
- [x] filter (array) (@ivoson)
- [ ] width_bucket
- [x] array_sort (@boneanxs)
- [ ] xpath
- [ ] xpath_boolean
- [ ] xpath_double
- [ ] xpath_float
- [ ] xpath_int
- [ ] xpath_long
- [ ] xpath_number
- [ ] xpath_short
- [ ] xpath_string
- [ ] unbase64 (WIP, @fyp711)
- [ ] decode (partially supported if translated to caseWhen. WIP Cody)
- [ ] initcap (WIP, velox PR: 8676)
- [x] unix_date (velox PR 8725, completed)
- [ ] count_min_sketch
- [x] bool_and/every (@mskapilks)
- [x] bool_or/any/some (@mskapilks)
- [x] shuffle (completed)
- [x] bround (@xumingming)
- [x] format_string (@gaoyangxiaozhu)
- [x] format_number (@gaoyangxiaozhu)
- [x] soundex (@zhli1142015)
- [x] levenshtein (@zhli1142015)
- [x] cot (@honeyhexin)
- [x] expm1 (@Donvi)
- [x] stack (generator function, @xumingming)
- [x] randn (@Donvi)
- [x] empty2null (internal function, @jinchengchenghh)
- [x] toprettystring (internal function, @jinchengchenghh)
- [x] AtLeastNNonNulls (internal funciton, @zhli1142015)
- [ ] GetStructField (internal funciton)
- [ ] [approx_count_distinct_for_intervals](https://github.com/apache/incubator-gluten/issues/11599)

* Since Spark-3.3 (related to ML, low priority)
- [ ] regr_count
- [ ] regr_avgx
- [ ] regr_avgy
- [x] regr_r2
- [ ] regr_sxx
- [x] regr_sxy
- [ ] regr_syy
- [ ] regr_slope
- [ ] regr_intercept

* Since Spark-3.3

* Since Spark-3.4
- [ ] mode
- [x] get (@Yohahaha)
- [x] array_append (@ivoson)
- [x] array_insert (@ivoson)
- [x] mode (@zhli1142015)

Contributor guide

Open the contributing guide

Research direction

Choose one unchecked function from the list, then review the Gluten function-support document and the corresponding Spark SQL references. Before starting, check the Velox sparksql and prestosql folders and any linked Velox work to avoid duplication; done means the selected function is supported and its status is reflected in the support tracking.

Written by the indexing model from the issue text.

Assessment

Tech stack
scala, sql
Domain
backend, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
28/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.