apache / apache/gluten

[VL] unix_timestamp reject non-ASCII (Unicode Nd) digits

Open
#12,825 1 comment 0 reactions 0 assignees View on GitHub
bug triage
Dominant language
Scala
Stars
1.6k
Forks
657
Avg merge
3d 17h
Merged PRs (30d)
86

Description

### Backend

VL (Velox)

### Bug description

`
SET spark.sql.optimizer.excludedRules=org.apache.spark.sql.catalyst.optimizer.ConstantFolding;
SELECT unix_timestamp('٢٠٢٦-٠٨-١٣ ٠٠:١٧:٤٨');
`
Vanilla Spark: 1786551468
Gluten + Velox: NULL

Velox's Spark-compatible date/time parsing only recognises ASCII 0-9. When the input string uses digits from another Unicode decimal-digit (Nd) block — e.g. Arabic-Indic U+0660-U+0669, Extended Arabic-Indic U+06F0-U+06F9, Devanagari U+0966-U+096F — parsing fails and the function returns NULL

### Gluten version

main branch

### Spark version

Spark-3.5.x

### Spark configurations

_No response_

### System information

_No response_

### Relevant logs

```bash

```

Contributor guide

Open the contributing guide

Research direction

Start by reproducing the unix_timestamp query with Arabic-Indic digits, then inspect Velox's Spark-compatible date/time parsing path. Compare its handling of Unicode Nd digits with the ASCII case and verify that the listed Arabic-Indic, Extended Arabic-Indic, and Devanagari inputs return the same result as Spark instead of NULL.

Written by the indexing model from the issue text.

Assessment

Tech stack
spark
Domain
data-engineering
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
58/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.