microsoft / microsoft/markitdown

Describing Images Inline in PDFs for Better RAG

Open
#1,346 2 comments 3 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

Title: Describing Images Inline in PDFs for Better RAG

I've been using LLMs to describe images like this, but its just support pass image directly:

from markitdown import MarkItDown
from openai import OpenAI

client = OpenAI()
md = MarkItDown(llm_client=client, llm_model="gpt-4o")
result = md.convert("example.jpg")
print(result.text_content)

However, in many cases, my PDF files contain both text and embedded images. I want to extract both and generate inline image descriptions in Markdown to improve retrieval quality in GenAI.

Is there a way to achieve something like this?

Original PDF content:

Lorem ipsum dolor sit amet...  
<-- image 1 - contains an orange cat --!>

Expected Markdown output:

Lorem ipsum dolor sit amet...

The image shows a fat orange cat sleeping on a white background.

How can I process both text and images together like this in a single conversion step?

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the repository’s PDF conversion path and comparing it with the documented image conversion example using MarkItDown and an LLM client. Determine how embedded images and surrounding text are currently handled; done should mean a single PDF conversion preserves the text and inserts generated image descriptions in the expected Markdown order.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
backend
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.