posit-dev / posit-dev/pointblank
Add row level comparisons to `Compare`
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 490
- Forks
- 32
- Avg merge
- 14h 24m
- Merged PRs (30d)
- 2
Description
- n unequal
- n equal
- n unique (probably helpful for diagnosing weird equality things)
This part will be difficult, there are 3 ways I think it can happen. First, is using some sort of exception logic like from a except b - blunt and useful. Second, using some row hashing functionality and then excepting the second from the first. Third would be using a join key(s), which I feel data will commonly have, and using one of the first 2 strategies from there, probably the hashing.
I've experimented on my own side in a professional setting with all of the 3. I find the exception method very easy but I strictly use duckdb for that. When I'm getting fancy or have some non-trivial join key logic I use the hashing but much to my dismay as it's usually categorically worse.
What I've had horrible experiences with is sorting join keys. I'm looking for a solution that can sort out of memory on quite large datasets, and I can't find a good solution that pre-sorts.
There are downsides to all these approaches and I think Narwhals will need some contributions in order to get this to work in a general sense. Polars has has hashing logic and duckdb has exception logic, can't speak for everything else.
Much to consider...
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by locating the existing Compare implementation and reviewing how the project uses Narwhals across supported backends. Clarify the intended behavior for unequal, equal, and unique rows, then compare the exception, hashing, and join-key approaches described in the issue. Done means an agreed general strategy and corresponding row-level comparison behavior.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- data, testing-qa
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100