Discussion: How to structure LLM, LMM blocks regarding structured usage of outputs in other blocks
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 2.5k
- Forks
- 320
- Avg merge
- 1d 14h
- Merged PRs (30d)
- 133
Description
Search before asking
- I have searched the Inference issues and found no similar feature requests.
Question
Some LLM/LMM would be able to produce output in specific format. For instance - detecting bounding boxes.
We have now LMM block and LMMForClassification block made solely for the purpose of producing structured output, required for compatibility with other blocks.
This approach is not scalable, we should think what to do.
Initial idea:
- LLM/LMM with predefined/configurable prompts causing specific output data to be created
- conversion blocks taking text outputs converting it into specific data (example: sv.Detections)
Additional
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified. Start by locating the existing LMM and LMMForClassification blocks and how their outputs are consumed by other blocks. Done would require an agreed, scalable design for structured LLM/LMM outputs or conversion blocks, but the issue does not define acceptance criteria.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- computer-vision, machine-learning
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100