[VL] Unsupported spark function list [please leave a comment if you plan to pick some]
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
Here listed spark functions still not supported by Gluten Velox backend. **Please leave a comment if you'd like to pick some.** In the below list, [√] means someone is working in progress for the corresponding function.
You can find all functions' support status from [this gluten doc](https://github.com/oap-project/gluten/blob/main/docs/velox-backend-support-progress.md#function-support).
To avoid duplicate work, before starting, please check whether a PR has been submitted in Velox community or whether it has already been implemented in Velox who holds most sql functions in its sparksql folder & prestosql folder.
### Reference:
* [spark sql expression ](https://github.com/apache/spark/blob/master/sql/core/src/test/resources/sql-functions/sql-expression-schema.md)
* [spark built-in functions](https://spark.apache.org/docs/latest/api/sql/index.html)
----------
- [x] percentile_approx/approx_percentile (WIP, guangxin)
- [x] concat_ws (PR ready, https://github.com/facebookincubator/velox/pull/8854)
- [x] unix_timestamp: "Only supports string type, with session timezone considered, todo: support date type"
- [x] locate
- [x] parse_url (PR drafted, not merged)
- [x] urldecoder: "UDF, supported by spark as a built-in function since 3.4.0."
- [ ] normalizenanandzero
- [x] arrayintersects
- [ ] default.json_split (udf, no need to impl.): "external UDF"
- [ ] parsejsonarray: "external UDF"
- [x] struct
- [x] percentile (@Yohahaha)
- [x] first/first_value (@JkSelf)
- [x] last/last_value (@JkSelf)
- [x] posexplode (WIP, @marin-ma)
- [x] trunc (WIP, HannanKan)
- [x] months_between (PR ready)
- [ ] stack
- [x] grouping_id
- [x] printf (@Surbhi-Vijay)
- [x] space (WIP, rhh777)
- [x] inline (WIP, @marin-ma)
- [x] to_unix_timestamp: "Only supports string type, with session timezone considered. todo: support date type"
- [x] from_csv (@mudit-97)
- [x] from_json (https://github.com/facebookincubator/velox/pull/11709)
- [x] to_json (@wecharyu)
- [x] json_object_keys (@dcoliversun)
- [ ] json_tuple
- [x] schema_of_csv
- [x] schema_of_json
- [x] to_csv
- [x] make_ym_interval (WIP, @marin-ma)
- [x] make_timestamp (WIP, @marin-ma)
- [x] make_interval
- [x] make_dt_interval
- [x] from_utc_timestamp (@acvictor)
- [ ] extract
- [ ] exists (@lyy-pineapple)
- [ ] date_part
- [ ] zip_with
- [x] transform (@Yohahaha)
- [ ] transform_keys
- [ ] transform_values
- [x] map_from_entries (WIP, MaYan)
- [x] map_filter (WIP, MaYan)
- [x] map_entries (Done, by MaYan)
- [x] forall (@lyy-pineapple)
- [x] flatten (@ivoson)
- [ ] filter
- [x] filter (array) (@ivoson)
- [ ] width_bucket
- [x] array_sort (@boneanxs)
- [ ] xpath
- [ ] xpath_boolean
- [ ] xpath_double
- [ ] xpath_float
- [ ] xpath_int
- [ ] xpath_long
- [ ] xpath_number
- [ ] xpath_short
- [ ] xpath_string
- [ ] unbase64 (WIP, @fyp711)
- [ ] decode (partially supported if translated to caseWhen. WIP Cody)
- [ ] initcap (WIP, velox PR: 8676)
- [x] unix_date (velox PR 8725, completed)
- [ ] count_min_sketch
- [x] bool_and/every (@mskapilks)
- [x] bool_or/any/some (@mskapilks)
- [x] shuffle (completed)
- [x] bround (@xumingming)
- [x] format_string (@gaoyangxiaozhu)
- [x] format_number (@gaoyangxiaozhu)
- [x] soundex (@zhli1142015)
- [x] levenshtein (@zhli1142015)
- [x] cot (@honeyhexin)
- [x] expm1 (@Donvi)
- [x] stack (generator function, @xumingming)
- [x] randn (@Donvi)
- [x] empty2null (internal function, @jinchengchenghh)
- [x] toprettystring (internal function, @jinchengchenghh)
- [x] AtLeastNNonNulls (internal funciton, @zhli1142015)
- [ ] GetStructField (internal funciton)
- [ ] [approx_count_distinct_for_intervals](https://github.com/apache/incubator-gluten/issues/11599)
* Since Spark-3.3 (related to ML, low priority)
- [ ] regr_count
- [ ] regr_avgx
- [ ] regr_avgy
- [x] regr_r2
- [ ] regr_sxx
- [x] regr_sxy
- [ ] regr_syy
- [ ] regr_slope
- [ ] regr_intercept
* Since Spark-3.3
* Since Spark-3.4
- [ ] mode
- [x] get (@Yohahaha)
- [x] array_append (@ivoson)
- [x] array_insert (@ivoson)
- [x] mode (@zhli1142015)
Contributor guide
Research direction
Choose one unchecked function from the list, then review the Gluten function-support document and the corresponding Spark SQL references. Before starting, check the Velox sparksql and prestosql folders and any linked Velox work to avoid duplication; done means the selected function is supported and its status is reflected in the support tracking.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- scala, sql
- Domain
- backend, data
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100