Support Levenshtein func with threshold parameter for Spark 4+
- Dominant language
- Rust
- Stars
- 1.8k
- Forks
- 241
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 21
Description
**Describe the bug**
The native `Levenshtein` function currently delegates to DataFusion's built-in `levenshtein`, which only supports 2 arguments (left, right). Starting from Spark 4, the `Levenshtein` expression supports an optional third `threshold` parameter, when the edit distance exceeds the threshold, it returns `-1` instead of the actual distance.
The native implementation falls back to DataFusion's `levenshtein` which ignores the threshold parameter, causing result mismatches with Spark on 3-argument calls.
**To Reproduce**
Because the native implementation lacked threshold support, the `"string Levenshtein distance"` test was excluded in Spark 4.0/4.1 test settings.
**Expected behavior**
- `levenshtein('kitten', 'sitting')` → `3`
- `levenshtein('kitten', 'sitting', 2)` → `-1` (distance 3 > threshold 2)
- `levenshtein('kitten', 'sitting', 5)` → `3` (distance 3 ≤ threshold 5)
**Screenshots**
**Additional context**
Contributor guide
Assessment
This issue has not been assessed yet.