posit-dev / posit-dev/pointblank

Add a method to get error_reports per_column in a dataframe

Open
#237 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: ★ Enhancement
Dominant language
Python
Stars
490
Forks
32
Avg merge
14h 24m
Merged PRs (30d)
2

Description

Prework

Proposal

I want to build an integration between pointblank validations and buckaroo. This will allow viewing the original dataframe inline, and highlighting errors. For reference Buckaroo is an interactive dataframe UI for notebooks, it lazily loads data into the browser and can scroll infinitely

To do this, I could use some help from pointblank.

Can you add a method that returns the original dataframe with "error_columns"

There could be a couple of approaches, but basically Buckaroo needs an additional column for each original column with a predictable name. This column should be null everywhere there isn't an error, when there is an error, just include the text explaining the error for that cell.

For pandas a multi-index of columns would be pretty cool, so make each column ('column_name', 'orig'), ('column_name', 'errors').

For polars you could make similar tuple columns.

Initially though just adding columns with a predictable name like 'column_name1__errors', 'column_name_foo__errors', will be easier for buckaroo. But that will inevitably run into some type of escaping error. Buckaroo can easily support this column name format right now. I'm very close to releasing multi-index column support and tuple column support.

What other cosiderations would go into a feature like that from your'alls side?

Here is a similar tool I built for pandera on top of Buckaroo
https://marimo.io/p/@paddy-mullen/buckaroo-pandera

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reviewing pointblank’s existing validation and error-reporting APIs, then compare the linked Buckaroo/Pandera integration. Clarify the pandas and polars output contract, including predictable error-column naming, null cells, and escaping. Done means an agreed API design with coverage for the supported dataframe backends.

Written by the indexing model from the issue text.

Assessment

Tech stack
pandas, python
Domain
data, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.