microsoft / microsoft/markitdown
Add support for Google's Gemini and Anthropic's Claude models
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 186k
- Forks
- 13.7k
- Avg merge
- 1d 4h
- Merged PRs (30d)
- 49
Description
Description
Currently, markitdown only supports OpenAI models for image captioning and content extraction. It would be valuable to add support for other leading multimodal LLMs, specifically:
- Google's Gemini models (Pro and Ultra)
- Anthropic's Claude models (Opus, Sonnet, and Haiku)
This would provide us with more flexibility and choice based on their preferences, API access, pricing, or specific model strengths.
Motivation
- Different users have access to different AI provider APIs
- Some users may prefer the strengths of a particular model family
- Pricing and rate limits vary between providers
- Organizations may have existing enterprise agreements with Google or Anthropic
Current implementation
Currently, _image_converter.py has hardcoded OpenAI-specific client API calls:
# Prepare the OpenAI API request
messages = [
{
"role": "user",
"content": [
{"type": "text", "text": prompt},
{
"type": "image_url",
"image_url": {
"url": data_uri,
},
},
],
}
]
# Call the OpenAI API
response = client.chat.completions.create(model=model, messages=messages)
return response.choices[0].message.content
Proposed solution
Create an abstraction layer for LLM providers that would:
- Detect the client type (OpenAI, Google, or Anthropic)
- Use the appropriate API format for each provider
- Extract the response content consistently
This could be implemented either:
- As provider-specific adapter classes
- Through a simple detection mechanism based on client type
- Or potentially by leveraging [Semantic Kernel](https://github.com/microsoft/markitdown/issues/232) as suggested in a related issue
Related issues
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with _image_converter.py and trace the current OpenAI-specific image captioning and content-extraction flow. Review related issues #232 and #12 before choosing between provider adapters, client detection, or Semantic Kernel. Done means Gemini and Claude models are supported alongside OpenAI with consistent response extraction.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ai
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100