posit-dev / posit-dev/chatlas

`content_image_url` throws error in local vision model

Open
#20 1 comment 1 reaction 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Python
Stars
176
Forks
28
Avg merge
18h 42m
Merged PRs (30d)
16

Description

Hi- thanks for creating this package! I'm testing out some local vision workflows with ollama and llama3.2-vision. I seem to get an error when using the content_image_url function to fetch an image:

from chatlas import ChatOllama, content_image_url, content_image_file

# Initialize chat with ollama llama3.2-vision
chat = ChatOllama(model="llama3.2-vision")

# Using content_image_url
chat.chat(
    "What do you see in this image?",
    content_image_url("https://www.python.org/static/img/python-logo.png"),
    echo="all"
)

Throws this error:



                                                           👤 User turn:

What do you see in this image?

<< 👤 other content >>

🌆 python-logo.png


                                                        🤖 Assistant turn:
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "\.venv\lib\site-packages\chatlas\_chat.py", line 356, in chat
    for _ in response:
  File "\.venv\lib\site-packages\chatlas\_chat.py", line 1169, in __next__
    chunk = next(self._generator)
  File "\.venv\lib\site-packages\chatlas\_chat.py", line 833, in _chat_impl
    for chunk in self._submit_turns(
  File "\.venv\lib\site-packages\chatlas\_chat.py", line 884, in _submit_turns
    response = self.provider.chat_perform(
  File "\.venv\lib\site-packages\chatlas\_openai.py", line 247, in chat_perform
    return self._client.chat.completions.create(**kwargs)  # type: ignore
  File "\.venv\lib\site-packages\openai\_utils\_utils.py", line 275, in wrapper
    return func(*args, **kwargs)
  File "\.venv\lib\site-packages\openai\resources\chat\completions.py", line 829, in create
    return self._post(
  File "\.venv\lib\site-packages\openai\_base_client.py", line 1280, in post
    return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
  File "\.venv\lib\site-packages\openai\_base_client.py", line 957, in request
    return self._request(
  File "\.venv\lib\site-packages\openai\_base_client.py", line 1061, in _request
    raise self._make_status_error_from_response(err.response) from None
openai.BadRequestError: Error code: 400 - {'error': {'message': 'invalid image input', 'type': 'invalid_request_error', 'param': None, 'code': None}}

The content_image_url("https://www.python.org/static/img/python-logo.png") seems to fetch the image and store it as a ContentImageRemote object.

Also, this works fine with content_image_file function:

chat.chat(
    "What do you see in this image?",
    content_image_file("python-logo.png"),
    echo="all"
)
                                👤 User turn:                                                                
                                                                                                             
What do you see in this image?                                                                               
                                                                                                             
<< 👤 other content >>                                                                                       
                                                                                                             
🌆 Dw2mJn+YhmuMAAAAAElFTkSuQmCC                                                                              
                                                                                                             
                                                                                                             
                              🤖 Assistant turn:                                                             
                                                                                                             
The image features the Python logo, which is a stylized representation of the                                
word "python" and serves as an emblem for the official implementation of the                                 
Python programming language.                                                                                 
                                                                                                             
Key Features:                                                                                                
                                                                                                             
 • Color scheme:                                                                                             
    • Primary color: Blue                                                                                    
    • Secondary color: Yellow                                                                                
 • Positioning: Left side of the screen                                                                      
                                                                                                             
Purpose: The logo is used to represent the official Python programming                                       
language, providing a clear visual identifier for users.    

I've tested this with version 0.2.0 and version 0.2.1.dev1+g6e81eb3 of chatlas.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with content_image_url and the ContentImageRemote path, then trace the request through chatlas/_chat.py and chatlas/_openai.py. Compare the remote-image request with the working content_image_file example and reproduce the Ollama llama3.2-vision call; done means the URL image is accepted without the invalid image input error.

Written by the indexing model from the issue text.

Assessment

Tech stack
ollama, python
Domain
ai
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
38/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.