Abraxas-365 / Abraxas-365/langchain-rust
Chain does not support images
- Linguagem predominante
- Rust
- Estrelas
- 1.3k
- Forks
- 177
- Métricas de merge de PRs
- Nenhum PR com merge em 30d
Descrição
**Describe the bug**
Using chain (`ConversationalChainBuilder` or `Agent`) does not support images.
```rust
prompt_args! {
"input" => input,
}
```
does not allow to pass an `image url`.
If the `url` is passed as a string the in input, an error is thrown `OpenAI error: stream failed: Invalid status code: 400 Bad Request` probably because the image is not under the right structure ([`Image content part`](https://platform.openai.com/docs/guides/vision)).
Also
`langchain_rust::schemas::Message::messages_to_string` only serialize text content. It does not take in consideration images.
https://github.com/Abraxas-365/langchain-rust/blob/891c56f0e2f2a98eed883f906c3de6cb82770797/src/schemas/messages.rs#L149-L155
**To Reproduce**
```rust
let chain = ConversationalChainBuilder::new()
.llm(llm)
.build();
let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
"input" => input,
});
```
**Expected behavior**
A way to pass image to a chain or at minimum an Agent.
Since images are on `langchain_rust::schemas::Message` structure, I would expect them to work in every aspect of the library.
**Additional context**
This is some kind of a violation of the `Liskov Principle` & `Interface Segregation Principles`. If image are available on `langchain_rust::schemas::Message` they should be supported everywhere in the library.
Guia de contribuição
Direção de pesquisa
The issue points to `langchain_rust::schemas::Message::messages_to_string` in src/schemas/messages.rs, which currently only serializes text content. Start by examining the Message struct to understand how image content is stored. Then, look at ConversationalChainBuilder and Agent to see how messages are processed and where image support needs to be added. Testing will involve creating a chain with an image URL and ensuring it conforms to OpenAI's vision API structure.
Escrita pelo modelo de indexação a partir do texto da issue.
Avaliação
- Stack de tecnologia
- rust
- Domínio
- ai-infra-agents
- Tipo de issue
- Bug
- Dificuldade
- 4/5
- Tempo estimado
- 3-5 dias
- Status de atividade
- Estagnada
- Clareza
- Claramente especificada
- Facilidade para iniciantes
- 45/100