[VL] unix_timestamp reject non-ASCII (Unicode Nd) digits
- Dominant language
- Scala
- Stars
- 1.6k
- Forks
- 657
- Avg merge
- 3d 17h
- Merged PRs (30d)
- 86
Description
### Backend
VL (Velox)
### Bug description
`
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SELECT unix_timestamp('٢٠٢٦-٠٨-١٣ ٠٠:١٧:٤٨');
`
Vanilla Spark: 1786551468
Gluten + Velox: NULL
Velox's Spark-compatible date/time parsing only recognises ASCII 0-9. When the input string uses digits from another Unicode decimal-digit (Nd) block — e.g. Arabic-Indic U+0660-U+0669, Extended Arabic-Indic U+06F0-U+06F9, Devanagari U+0966-U+096F — parsing fails and the function returns NULL
### Gluten version
main branch
### Spark version
Spark-3.5.x
### Spark configurations
_No response_
### System information
_No response_
### Relevant logs
```bash
```
Contributor guide
Research direction
Start by reproducing the unix_timestamp query with Arabic-Indic digits, then inspect Velox's Spark-compatible date/time parsing path. Compare its handling of Unicode Nd digits with the ASCII case and verify that the listed Arabic-Indic, Extended Arabic-Indic, and Devanagari inputs return the same result as Spark instead of NULL.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- spark
- Domain
- data-engineering
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 58/100