microsoft / microsoft/markitdown

Add support for Google's Gemini and Anthropic's Claude models

Open
#1,129 9 comments 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
186k
Forks
13.7k
Avg merge
1d 4h
Merged PRs (30d)
49

Description

Description

Currently, markitdown only supports OpenAI models for image captioning and content extraction. It would be valuable to add support for other leading multimodal LLMs, specifically:

  1. Google's Gemini models (Pro and Ultra)
  2. Anthropic's Claude models (Opus, Sonnet, and Haiku)

This would provide us with more flexibility and choice based on their preferences, API access, pricing, or specific model strengths.

Motivation

  • Different users have access to different AI provider APIs
  • Some users may prefer the strengths of a particular model family
  • Pricing and rate limits vary between providers
  • Organizations may have existing enterprise agreements with Google or Anthropic

Current implementation

Currently, _image_converter.py has hardcoded OpenAI-specific client API calls:

# Prepare the OpenAI API request
messages = [
    {
        "role": "user",
        "content": [
            {"type": "text", "text": prompt},
            {
                "type": "image_url",
                "image_url": {
                    "url": data_uri,
                },
            },
        ],
    }
]

# Call the OpenAI API
response = client.chat.completions.create(model=model, messages=messages)
return response.choices[0].message.content

Proposed solution

Create an abstraction layer for LLM providers that would:

  1. Detect the client type (OpenAI, Google, or Anthropic)
  2. Use the appropriate API format for each provider
  3. Extract the response content consistently

This could be implemented either:

Related issues

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with _image_converter.py and trace the current OpenAI-specific image captioning and content-extraction flow. Review related issues #232 and #12 before choosing between provider adapters, client detection, or Semantic Kernel. Done means Gemini and Claude models are supported alongside OpenAI with consistent response extraction.

Written by the indexing model from the issue text.

Assessment

Tech stack
python
Domain
ai
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Active
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.