apache / apache/datafusion

bug: datafusion-spark string literals don't interpret escape sequences like Spark

Open
#21,516 5 comments 0 reactions 0 assignees View on GitHub
spark
Dominant language
Rust
Stars
9.3k
Forks
2.4k
Avg merge
3d 7h
Merged PRs (30d)
344

Description

### Describe the bug

DataFusion-spark treats `\t` and `\n` in SQL string literals as literal backslash characters, while Apache Spark interprets them as escape sequences (tab and newline). This affects any function that receives string arguments containing these sequences.

### To Reproduce

**PySpark (Spark behavior):**
```sql
SELECT soundex('\thello'); -- returns tab + "hello" (soundex passes through non-alpha input)
SELECT soundex('\nhello'); -- returns newline + "hello"
SELECT length('\thello'); -- 6 (tab is one character)
SELECT length('\nhello'); -- 6 (newline is one character)
```

**DataFusion-spark (current behavior):**
```sql
SELECT soundex('\thello'); -- returns literal "\thello" (backslash-t-hello)
SELECT soundex('\nhello'); -- returns literal "\nhello" (backslash-n-hello)
SELECT length('\thello'); -- 7 (\t is two characters: backslash and t)
SELECT length('\nhello'); -- 7 (\n is two characters: backslash and n)
```

### Expected behavior

DataFusion-spark should interpret `\t`, `\n`, and other escape sequences in string literals the same way Spark does, for Spark compatibility.

### Additional context

This is a string literal parsing issue, not specific to `soundex`. It affects all string functions. The `.slt` tests at `string/soundex.slt` lines 83 and 193 have expected values that match DataFusion's literal interpretation rather than Spark's escape interpretation.

This was discovered by running a PySpark validation script against the `.slt` test files (see #17045, #21508).

Contributor guide

Open the contributing guide

Research direction

Start with the string literal parsing path and inspect string/soundex.slt at lines 83 and 193, where the expected values reflect DataFusion's current interpretation. Compare the cases with the PySpark examples and update parsing and tests so \t, \n, and other supported escapes match Spark across string functions.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust, sql
Domain
databases
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
68/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.