Feature Request: Support multimodal extraction (text + image)
- Dominant language
- Python
- Stars
- 38.6k
- Forks
- 2.7k
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 3
Description
It would be great if langextract could support multimodal input, allowing users to pass both textual and visual data directly to the model without pre-processing images via VL model. This would leverage modern multimodal models capable of handling image understanding tasks together with text.
Currently, when working with image-rich documents or visual datasets, we need to run VL model to convert images to text before extraction. This loses potential visual context informationand adds extra processing steps. Many recent LLMs can directly accept images alongside text prompts.
Supporting this natively in langextract would:
- Save processing time
- Preserve visual context
- Enable richer extraction capabilities from images
1. Is there any plan to support multimodal extraction (text + image) in langextract?
2. Do you have any recommended best practices or existing approaches for this scenario?
3. If there’s no current plan, would you welcome community contributions for such a feature?
Contributor guide
Research direction
No files, tests, or entry points are named. Start by reviewing the current extraction interface and model integrations to determine how text and images should be represented; done means an agreed multimodal API and documented behavior for supported models.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100