microsoft / microsoft/markitdown
Describing Images Inline in PDFs for Better RAG
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 186k
- Forks
- 13.7k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 49
Description
Title: Describing Images Inline in PDFs for Better RAG
I've been using LLMs to describe images like this, but its just support pass image directly:
from markitdown import MarkItDown
from openai import OpenAI
client = OpenAI()
md = MarkItDown(llm_client=client, llm_model="gpt-4o")
result = md.convert("example.jpg")
print(result.text_content)
However, in many cases, my PDF files contain both text and embedded images. I want to extract both and generate inline image descriptions in Markdown to improve retrieval quality in GenAI.
Is there a way to achieve something like this?
Original PDF content:
Lorem ipsum dolor sit amet...
<-- image 1 - contains an orange cat --!>
Expected Markdown output:
Lorem ipsum dolor sit amet...
The image shows a fat orange cat sleeping on a white background.
How can I process both text and images together like this in a single conversion step?
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by tracing the repository’s PDF conversion path and comparing it with the documented image conversion example using MarkItDown and an LLM client. Determine how embedded images and surrounding text are currently handled; done should mean a single PDF conversion preserves the text and inserts generated image descriptions in the expected Markdown order.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- backend
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100