huggingface / huggingface/chat-ui

IDEFICS works with image URLs but not "upload image"

Open
#715 4 comments 0 reactions 0 assignees View on GitHub
bug models
Dominant language
TypeScript
Stars
11k
Forks
1.7k
Avg merge
21h 24m
Merged PRs (30d)
56

Description

Hi there,

I am hosting IDEFICS on TGI locally and hitting it via Chat-UI. When I go to a URL, it is able to correctly work with images, as seen here:
image

However, when I use the "upload image" feature, or the drag and drop, it looks like it is ignoring the image:
image

Here is my model config (using the latest main branch of chat-ui 0.7.0):
` {
"name": "HuggingFaceM4/idefics-9b-instruct",
"endpoints":[{"type":"tgi","url":"http://INTERNAL URL:/generate_stream"}],
"multimodal" : true,
"description": "IDEFICS is the new multimodal model by Hugging Face.",
"preprompt": "",
"chatPromptTemplate" : "{{#each messages}}{{#ifUser}}User: {{content}}{{/ifUser}}\nAssistant: {{#ifAssistant}}{{content}}\n{{/ifAssistant}}{{/each}}",
"parameters": {
"temperature": 0.1,
"top_p": 0.95,
"repetition_penalty": 1.2,
"top_k": 12,
"truncate": 1000,
"max_new_tokens": 1024,
"stop": ["", "User:", "\nUser:"]
}
}`

And here is my TGI command args from docker compose (using 1.3.4 container for TGI):
` --sharded true --num-shard 2 --dtype float16 --model-id HuggingFaceM4/idefics-9b-instruct --max-total-tokens 2048 --max-input-length 1000 --max-batch-prefill-tokens 2048`

Here is the [image URL](https://commons.wikimedia.org/wiki/Dog#/media/File:Flat_Coated_Retriever_-_black.jpg) for reference.

A gigantic thank you for putting together such wonderful tools!! And thank you in advance for your help.

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.