google / google/langextract

Feature Request: Support multimodal extraction (text + image)

Open
#270 3 comments 10 reactions 0 assignees View on GitHub
discussion
Dominant language
Python
Stars
38.6k
Forks
2.7k
Avg merge
3d 15h
Merged PRs (30d)
3

Description

It would be great if langextract could support multimodal input, allowing users to pass both textual and visual data directly to the model without pre-processing images via VL model. This would leverage modern multimodal models capable of handling image understanding tasks together with text.

Currently, when working with image-rich documents or visual datasets, we need to run VL model to convert images to text before extraction. This loses potential visual context informationand adds extra processing steps. Many recent LLMs can directly accept images alongside text prompts.
Supporting this natively in langextract would:
- Save processing time
- Preserve visual context
- Enable richer extraction capabilities from images

1. Is there any plan to support multimodal extraction (text + image) in langextract?
2. Do you have any recommended best practices or existing approaches for this scenario?
3. If there’s no current plan, would you welcome community contributions for such a feature?

Contributor guide

Open the contributing guide

Research direction

No files, tests, or entry points are named. Start by reviewing the current extraction interface and model integrations to determine how text and images should be represented; done means an agreed multimodal API and documented behavior for supported models.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai, machine-learning
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.