apache / apache/auron

Support Levenshtein func with threshold parameter for Spark 4+

Open
#2,279 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Rust
Stars
1.8k
Forks
241
Avg merge
2d 12h
Merged PRs (30d)
21

Description

**Describe the bug**

The native `Levenshtein` function currently delegates to DataFusion's built-in `levenshtein`, which only supports 2 arguments (left, right). Starting from Spark 4, the `Levenshtein` expression supports an optional third `threshold` parameter, when the edit distance exceeds the threshold, it returns `-1` instead of the actual distance.

The native implementation falls back to DataFusion's `levenshtein` which ignores the threshold parameter, causing result mismatches with Spark on 3-argument calls.

**To Reproduce**

Because the native implementation lacked threshold support, the `"string Levenshtein distance"` test was excluded in Spark 4.0/4.1 test settings.

**Expected behavior**

- `levenshtein('kitten', 'sitting')` → `3`
- `levenshtein('kitten', 'sitting', 2)` → `-1` (distance 3 > threshold 2)
- `levenshtein('kitten', 'sitting', 5)` → `3` (distance 3 ≤ threshold 5)

**Screenshots**

**Additional context**

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.