posit-dev / posit-dev/pointblank

Add row level comparisons to `Compare`

Open
#91 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: ★ Enhancement
Dominant language
Python
Stars
490
Forks
32
Avg merge
14h 24m
Merged PRs (30d)
2

Description

  • n unequal
  • n equal
  • n unique (probably helpful for diagnosing weird equality things)

This part will be difficult, there are 3 ways I think it can happen. First, is using some sort of exception logic like from a except b - blunt and useful. Second, using some row hashing functionality and then excepting the second from the first. Third would be using a join key(s), which I feel data will commonly have, and using one of the first 2 strategies from there, probably the hashing.

I've experimented on my own side in a professional setting with all of the 3. I find the exception method very easy but I strictly use duckdb for that. When I'm getting fancy or have some non-trivial join key logic I use the hashing but much to my dismay as it's usually categorically worse.

What I've had horrible experiences with is sorting join keys. I'm looking for a solution that can sort out of memory on quite large datasets, and I can't find a good solution that pre-sorts.

There are downsides to all these approaches and I think Narwhals will need some contributions in order to get this to work in a general sense. Polars has has hashing logic and duckdb has exception logic, can't speak for everything else.

Much to consider...

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by locating the existing Compare implementation and reviewing how the project uses Narwhals across supported backends. Clarify the intended behavior for unequal, equal, and unique rows, then compare the exception, hashing, and join-key approaches described in the issue. Done means an agreed general strategy and corresponding row-level comparison behavior.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
data, testing-qa
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.