[VL] Add ANSI mode support
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 2d 14h
- Merged PRs (30d)
- 80
Description
### Description
In this issue, we will list and track all tasks for ANSI mode support.
There are two Spark configurations directly related to ANSI usage.
* `spark.sql.ansi.enabled` (default is true since Spark 4.0)
* `spark.sql.storeAssignmentPolicy` (default is ANSI since Spark 3.0)
Please note this task list may not be complete; suggestions for additional tasks are welcome. Checked tasks have been claimed and may already have an associated PR or be finished. You are welcome to take on the unchecked tasks, but before doing so, please verify against the Velox codebase whether a relevant PR has already been created or merged.
## Basic
- [x] https://github.com/apache/incubator-gluten/issues/10387
## **1. Type Casting Functions (ANSI Strict)**
- [x] cast string to boolean (**FINISHED**)
[Cast.scala#L701](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L701)
- [x] Cast decimal to string (**FINISHED**): https://github.com/facebookincubator/velox/pull/16124
Follow-up PR: https://github.com/facebookincubator/velox/pull/17647
// In ANSI mode, Spark always use plain string representation on casting Decimal values
// as strings. Otherwise, the casting is using `BigDecimal.toString` which may use scientific
// notation if an exponent is needed.
[Cast.scala#L678](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L678)
- [x] cast string to timestamp (@infvg) [**FINISHED**]: https://github.com/facebookincubator/velox/pull/17102
[Cast.scala#L733](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L733)
- [ ] cast String to timestampNTZ
[Cast.scala#L775](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L775)
- [x] cast float/double to timestamp (@infvg) [**FINISHED**]: https://github.com/facebookincubator/velox/pull/17219
[Cast.scala#L758](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L758)
[Cast.scala#L765](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L765)
- [x] cast string to date (**FINISHED**): https://github.com/facebookincubator/velox/pull/16092
[Cast.scala#L811](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L811)
- [x] cast string to time (@malinjawi) [**FINISHED**]:: https://github.com/facebookincubator/velox/pull/16123
[Cast.scala#L826](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L826)
The implementation for codegen, assume equivalent with the above link:
[Cast.scala#L1493](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1493)
- [x] cast string to long/int/short/byte (**FINISHED**)
As one example, here is the related code for long type:
[Cast.scala#L883](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L883)
- [x] cast NumericType to long/int/short/byte (@minni31) [**FINISHED**]:: https://github.com/facebookincubator/velox/pull/16962
As one example, here is the related code for long type:
[Cast.scala#L896](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L896)
- [x] cast timestamp to int/short/byte (**FINISHED**): https://github.com/facebookincubator/velox/pull/17977
As one example, here is the related code for int type:
[Cast.scala#L933](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L933)
- [ ] cast time to short/byte (requires TimeType support)
As one example, here is the related code for short type:
[Cast.scala#L980](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L980)
- [x] cast string to double/float: https://github.com/facebookincubator/velox/pull/17285
ANSI controls the behavior in handling incorrect number format
As one example, here is the related code for double type:
[Cast.scala#L1159](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1159)
**cast several types to decimal**
ANSI controls the overflow behavior in changePrecision
[Cast.scala#L1103](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/Cast.scala#L1103)
- [x] cast VARCHAR to decimal [**FINISHED**]: https://github.com/facebookincubator/velox/pull/18019
- [x] cast INTEGRAL to decimal [**FINISHED**]: https://github.com/facebookincubator/velox/pull/18107
- [x] fix: Make Spark CAST(floating-point as decimal) ANSI-compliant [**FINISHED**]: https://github.com/facebookincubator/velox/pull/18206
- [x] fix(spark): Enable ANSI-compliant cast from DECIMAL to DECIMAL [**FINISHED**]: https://github.com/facebookincubator/velox/pull/18415
- [x] fix(spark): Enable ANSI-compliant cast from BOOLEAN to DECIMAL [**FINISHED**]: https://github.com/facebookincubator/velox/pull/18510
## **2. Math Functions (ANSI Overflow Check)**
- [x] A base type: AnsiIntervalType (@malinjawi): https://github.com/facebookincubator/velox/pull/17098
[AbstractDataType.scala#L168](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/api/src/main/scala/org/apache/spark/sql/types/AbstractDataType.scala#L168)
Unary expressions like Abs, UnaryMinus
The ANSI config controls failOnError.
[arithmetic.scala#L152C35-L152C46](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L152C35-L152C46)
- [ ] Abs
- [x] UnaryMinus [**FINISHED**]:
https://github.com/facebookincubator/velox/pull/16361
https://github.com/facebookincubator/velox/pull/16672
https://github.com/facebookincubator/velox/pull/18096
Binary arithmetic expressions using BinaryArithmetic as base, such as add, divide, multiply, remainder, pmod, etc., for all supported types (@malinjawi)
[arithmetic.scala#L209](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/arithmetic.scala#L209)
- [x] add/subtract (decimal type): https://github.com/facebookincubator/velox/pull/16302
- [x] divide (decimal type): https://github.com/facebookincubator/velox/pull/16323
- [x] multiply (decimal type)(**FINISHED**): https://github.com/facebookincubator/velox/pull/16307
- [x] add/subtract/divide/multiply (other types): https://github.com/facebookincubator/velox/pull/16361
- [x] remainder (integral types and floating point types): https://github.com/facebookincubator/velox/pull/16403
- [x] pmod (integral types and floating point types): https://github.com/facebookincubator/velox/pull/16405
- [x] pmod (decimal type): https://github.com/facebookincubator/velox/pull/16406
round functions
As one example, see how to round to ByteType with ANSI enabled:
[mathExpressions.scala#L1579](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L1579)
- [x] Round: https://github.com/facebookincubator/velox/pull/16479
- [ ] BRound
- [ ] RoundCeil
- [ ] RoundFloor
## **3. Date/Time Functions (ANSI Validation)**
- [x] ToUnixTimestamp: https://github.com/facebookincubator/velox/pull/18765
https://github.com/apache/spark/blob/9e0bea0c104a558d798706675a9bb47612a0575b/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/datetimeExpressions.scala#L1882
- [x] UnixTimestamp: https://github.com/facebookincubator/velox/pull/18765
- [ ] GetTimestamp
- [ ] TryToTimestampExpressionBuilder
- [ ] NextDay
- [ ] DateAddInterval
- [ ] ParseToDate
- [ ] TryToDateExpressionBuilder
- [ ] ParseToTimestamp
- [x] MakeDate: https://github.com/facebookincubator/velox/pull/18764
- [ ] TryMakeTimestampLTZExpressionBuilder
- [x] MakeTimestamp: https://github.com/facebookincubator/velox/pull/17745
## **4. Misc**
- [ ] String expressions: Elt
[stringExpressions.scala#L286](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/stringExpressions.scala#L286)
- [x] Collection expression Size: https://github.com/facebookincubator/velox/pull/18822
Its legacySizeOfNull is impacted by ANSI config.
[collectionOperations.scala#L118](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L118)
- [ ] Collection expression ElementAt
[collectionOperations.scala#L2622](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/collectionOperations.scala#L2622)
- [ ] conv [mathExpressions.scala#L451](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/mathExpressions.scala#L451)
- [x] STORE_ASSIGNMENT_POLICY defaults to ANSI [**FINISHED**]: https://github.com/apache/gluten/pull/12051
[SQLConf.scala#L4487](https://github.com/apache/spark/blob/e221b56be7b6d9e48e107fc4d1cf0c15f02700f8/sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala#L4487)
## 5. **Aggregation Functions (ANSI Overflow)**
### `SUM`, `AVG`, `VAR_POP`, `VAR_SAMP`, `STDDEV_POP`, `STDDEV_SAMP`
- In ANSI mode: overflow checks during accumulation.
- [x] SUM: https://github.com/facebookincubator/velox/pull/16435
- [x] AVG: https://github.com/facebookincubator/velox/pull/16439
- [ ] VAR_POP
- [ ] VAR_SAMP
- [ ] STDDEV_POP
- [ ] STDDEV_SAMP
### `TRY_SUM` (Spark 3.4+)
- Returns `NULL` on overflow instead of error.
## 6. **Window Functions (ANSI Overflow)**
Same overflow checks apply in window operations:
- `SUM(...) OVER(...)`
- `AVG(...) OVER(...)`
## 7. **ANSI SQL Compliant String Functions**
### `SUBSTRING` / `SUBSTR`
- ANSI SQL standard argument order: `SUBSTRING(str FROM start [FOR len])`
- Also supports classic form: `SUBSTRING(str, start, len)`
### `TRIM`
- ANSI syntax: `TRIM(LEADING '0' FROM col)` Also `TRIM(BOTH ...)`, `TRIM(TRAILING ...)`
### `OVERLAY`
- ANSI SQL string replacement: `OVERLAY(string PLACING replacement FROM start [FOR length])`
---
See Spark ANSI compliance: https://github.com/apache/spark/blob/v4.0.0/docs/sql-ref-ansi-compliance.md
Related discussion: https://github.com/apache/incubator-gluten/issues/4740.
https://github.com/facebookincubator/velox/issues/3869
Contributor guide
Assessment
This issue has not been assessed yet.