Key retraction misfires on deletion-heavy diffs: weigh matched-row agreement
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 27
- Forks
- 0
- Avg merge
- 53m
- Merged PRs (30d)
- 1
Description
Problem
When a guessed key is correct but the diff is dominated by row deletions, change_mass counts dropped rows in full, pushing changed mass past MAX_PLAUSIBLE_CHANGE_PERCENT (50%). The correct key is then retracted as excessive_change, and the second guess can land on a structurally-unique but semantically wrong column.
Reproducer from the SDVB benchmark (IMDB mixed_mixed_removal_rows2, 800 → 47 rows, a genuine 94% row deletion):
# declared key: honest answer
table_key([Series_Title], basis: declared)
row_drop(rows: 753)
# guessed key: retraction misfires
key_retracted([Series_Title], reason: excessive_change)
----
table_key([Overview], basis: guessed, overlap: 1.00)
table_regenerate()
The retracted key was producing the better diff; the re-guess on Overview (free text, unique on both sides) yields table_regenerate() — strictly less informative.
Proposed signal
The two cases the retraction trigger must distinguish leave different evidence in pass one:
- Wrong key: matched rows disagree heavily in their cells.
- Right key + massive deletion: matched rows are essentially clean; the change mass comes entirely from dropped rows.
In the example above, the 47 matched rows under Series_Title had ~zero changed cells, fully corroborating the key. A refinement: retract only when the matched rows themselves disagree heavily, or exempt deletion-dominated mass when matched rows are clean.
Found while running data-diff against the SDVB benchmark; also affects the WINE_small key_retracted cases (3 pairs).
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the SDVB IMDB mixed_mixed_removal_rows2 case and trace the pass-one key-retraction logic around change_mass and matched-row agreement. Done means deletion-dominated diffs with clean matched rows preserve the correct key, while keys whose matched rows disagree heavily can still be retracted; check the WINE_small cases as regression coverage.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- data
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 50/100