posit-dev / posit-dev/pointblank

Multimodal inputs in the `DraftValidation` class

Open
#394 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Type: ★ Enhancement
Dominant language
Python
Stars
490
Forks
32
Avg merge
14h 24m
Merged PRs (30d)
2

Description

Prework

Proposal

Continuing on the theme of multimodal inputs, I thought it'd be really cool if the DraftValidation class could support multimodal inputs (images and PDF), just as Validate.prompt() now does. A rough sketch of how that would work, drawing inspiration from the How DraftValidation Works steps:

  1. A statistical summary of your data is generated using the DataScan class.
  2. This summary is converted to JSON format and sent to your selected LLM provider.
  3. NEW: You provide the DraftValidation class with an image or PDF file describing what the data should look like. For example, the image could be a schema diagram, while the PDF could outline the project configuration elements that the data should conform to.
  4. The LLM uses the summary and multimodal input along with knowledge about Pointblank’s validation capabilities to generate a validation plan.
  5. The result is returned as executable Python code that you can use directly or modify as needed

Happy to get feedback from you @rich-iannone, and iterate. I'd love to implement this.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reading the DraftValidation and Validate.prompt() entry points, then review DataScan and the linked How DraftValidation Works guide to understand the existing flow. Done means defining and implementing support for image or PDF inputs so the LLM can use them with the data summary to produce executable validation code.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, data
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.