vercel-labs / vercel-labs/knowledge-agent-template
Add Image File Support as a Knowledge Source
Nobody has claimed this yet.
- Dominant language
- TypeScript
- Stars
- 1k
- Forks
- 131
- PR merge metrics
- No merged PRs in 30d
Description
Description
Feature Request: Image File Support in Knowledge Sources
Problem
Currently, the knowledge agent only supports text-based sources (GitHub repos,
YouTube transcripts, custom APIs). Many real-world knowledge bases contain
valuable information embedded in images — diagrams, charts, screenshots,
scanned documents, and architecture diagrams — which the agent cannot process.
Proposed Solution
Add support for image files (.png, .jpg, .jpeg, .webp, .gif) as a
knowledge source type, enabling the agent to:
- Ingest images from configured sources (GitHub repo folders, URLs, uploads)
- Extract text/content via OCR or a multimodal LLM
- Store extracted content in the sandbox snapshot so existing
grep/cat
tools can search it like any other text file - Preserve image metadata (filename, alt text, captions) alongside content
Use Cases
- Searching architecture diagrams or flowcharts stored as images
- Indexing scanned PDFs or documentation screenshots
- Querying infographics and charts containing embedded text
- Supporting docs where visuals are the primary communication medium
Additional context
No response
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
No files, tests, or entry points are identified in the issue. Start by tracing how current GitHub repositories, YouTube transcripts, and custom APIs become sandbox snapshot content, then determine where image ingestion, OCR or multimodal extraction, and metadata preservation fit. Done means supported image sources can be searched through the existing text tools with their metadata retained.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- typescript
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100