bug: datafusion-spark string literals don't interpret escape sequences like Spark
- Dominant language
- Rust
- Stars
- 9.3k
- Forks
- 2.4k
- Avg merge
- 3d 7h
- Merged PRs (30d)
- 344
Description
### Describe the bug
DataFusion-spark treats `\t` and `\n` in SQL string literals as literal backslash characters, while Apache Spark interprets them as escape sequences (tab and newline). This affects any function that receives string arguments containing these sequences.
### To Reproduce
**PySpark (Spark behavior):**
```sql
SELECT soundex('\thello'); -- returns tab + "hello" (soundex passes through non-alpha input)
SELECT soundex('\nhello'); -- returns newline + "hello"
SELECT length('\thello'); -- 6 (tab is one character)
SELECT length('\nhello'); -- 6 (newline is one character)
```
**DataFusion-spark (current behavior):**
```sql
SELECT soundex('\thello'); -- returns literal "\thello" (backslash-t-hello)
SELECT soundex('\nhello'); -- returns literal "\nhello" (backslash-n-hello)
SELECT length('\thello'); -- 7 (\t is two characters: backslash and t)
SELECT length('\nhello'); -- 7 (\n is two characters: backslash and n)
```
### Expected behavior
DataFusion-spark should interpret `\t`, `\n`, and other escape sequences in string literals the same way Spark does, for Spark compatibility.
### Additional context
This is a string literal parsing issue, not specific to `soundex`. It affects all string functions. The `.slt` tests at `string/soundex.slt` lines 83 and 193 have expected values that match DataFusion's literal interpretation rather than Spark's escape interpretation.
This was discovered by running a PySpark validation script against the `.slt` test files (see #17045, #21508).
Contributor guide
Research direction
Start with the string literal parsing path and inspect string/soundex.slt at lines 83 and 193, where the expected values reflect DataFusion's current interpretation. Compare the cases with the PySpark examples and update parsing and tests so \t, \n, and other supported escapes match Spark across string functions.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust, sql
- Domain
- databases
- Issue type
- Bug
- Difficulty
- 3/5
- Estimated time
- 1-2 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 68/100