Abraxas-365 / Abraxas-365/langchain-rust

Chain does not support images

Aberta
#238 0 comentários 0 reações 0 responsáveis Ver no GitHub
Linguagem predominante
Rust
Estrelas
1.3k
Forks
177
Métricas de merge de PRs
Nenhum PR com merge em 30d

Descrição

**Describe the bug**
Using chain (`ConversationalChainBuilder` or `Agent`) does not support images.

```rust
prompt_args! {
"input" => input,
}
```

does not allow to pass an `image url`.
If the `url` is passed as a string the in input, an error is thrown `OpenAI error: stream failed: Invalid status code: 400 Bad Request` probably because the image is not under the right structure ([`Image content part`](https://platform.openai.com/docs/guides/vision)).

Also
`langchain_rust::schemas::Message::messages_to_string` only serialize text content. It does not take in consideration images.
https://github.com/Abraxas-365/langchain-rust/blob/891c56f0e2f2a98eed883f906c3de6cb82770797/src/schemas/messages.rs#L149-L155

**To Reproduce**
```rust
let chain = ConversationalChainBuilder::new()
.llm(llm)
.build();

let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
"input" => input,
});
```

**Expected behavior**
A way to pass image to a chain or at minimum an Agent.

Since images are on `langchain_rust::schemas::Message` structure, I would expect them to work in every aspect of the library.

**Additional context**
This is some kind of a violation of the `Liskov Principle` & `Interface Segregation Principles`. If image are available on `langchain_rust::schemas::Message` they should be supported everywhere in the library.

Guia de contribuição

Abrir o guia de contribuição

Direção de pesquisa

The issue points to `langchain_rust::schemas::Message::messages_to_string` in src/schemas/messages.rs, which currently only serializes text content. Start by examining the Message struct to understand how image content is stored. Then, look at ConversationalChainBuilder and Agent to see how messages are processed and where image support needs to be added. Testing will involve creating a chain with an image URL and ensuring it conforms to OpenAI's vision API structure.

Escrita pelo modelo de indexação a partir do texto da issue.

Avaliação

Stack de tecnologia
rust
Domínio
ai-infra-agents
Tipo de issue
Bug
Dificuldade
4/5
Tempo estimado
3-5 dias
Status de atividade
Estagnada
Clareza
Claramente especificada
Facilidade para iniciantes
45/100

Receba novas issues na sua caixa de entrada

Um resumo curto de issues do GitHub para quem está começando.