vercel-labs / vercel-labs/knowledge-agent-template

Add Image File Support as a Knowledge Source

Open
#45 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

enhancement
Dominant language
TypeScript
Stars
1k
Forks
131
PR merge metrics
No merged PRs in 30d

Description

Description

Feature Request: Image File Support in Knowledge Sources

Problem

Currently, the knowledge agent only supports text-based sources (GitHub repos,
YouTube transcripts, custom APIs). Many real-world knowledge bases contain
valuable information embedded in images — diagrams, charts, screenshots,
scanned documents, and architecture diagrams — which the agent cannot process.

Proposed Solution

Add support for image files (.png, .jpg, .jpeg, .webp, .gif) as a
knowledge source type, enabling the agent to:

  1. Ingest images from configured sources (GitHub repo folders, URLs, uploads)
  2. Extract text/content via OCR or a multimodal LLM
  3. Store extracted content in the sandbox snapshot so existing grep/cat
    tools can search it like any other text file
  4. Preserve image metadata (filename, alt text, captions) alongside content
Use Cases
  • Searching architecture diagrams or flowcharts stored as images
  • Indexing scanned PDFs or documentation screenshots
  • Querying infographics and charts containing embedded text
  • Supporting docs where visuals are the primary communication medium
Additional context

No response

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

No files, tests, or entry points are identified in the issue. Start by tracing how current GitHub repositories, YouTube transcripts, and custom APIs become sandbox snapshot content, then determine where image ingestion, OCR or multimodal extraction, and metadata preservation fit. Done means supported image sources can be searched through the existing text tools with their metadata retained.

Written by the indexing model from the issue text.

Assessment

Tech stack
typescript
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.