Abraxas-365 / Abraxas-365/langchain-rust

Chain does not support images

Abierto
#238 0 comentarios 0 reacciones 0 asignados Ver en GitHub
Lenguaje dominante
Rust
Estrellas
1.3k
Forks
177
Métricas de merge de PR
Sin PR fusionados en 30 d

Descripción

**Describe the bug**
Using chain (`ConversationalChainBuilder` or `Agent`) does not support images.

```rust
prompt_args! {
"input" => input,
}
```

does not allow to pass an `image url`.
If the `url` is passed as a string the in input, an error is thrown `OpenAI error: stream failed: Invalid status code: 400 Bad Request` probably because the image is not under the right structure ([`Image content part`](https://platform.openai.com/docs/guides/vision)).

Also
`langchain_rust::schemas::Message::messages_to_string` only serialize text content. It does not take in consideration images.
https://github.com/Abraxas-365/langchain-rust/blob/891c56f0e2f2a98eed883f906c3de6cb82770797/src/schemas/messages.rs#L149-L155

**To Reproduce**
```rust
let chain = ConversationalChainBuilder::new()
.llm(llm)
.build();

let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
"input" => input,
});
```

**Expected behavior**
A way to pass image to a chain or at minimum an Agent.

Since images are on `langchain_rust::schemas::Message` structure, I would expect them to work in every aspect of the library.

**Additional context**
This is some kind of a violation of the `Liskov Principle` & `Interface Segregation Principles`. If image are available on `langchain_rust::schemas::Message` they should be supported everywhere in the library.

Guía de contribución

Abrir la guía de contribución

Evaluación

Este issue todavía no se ha evaluado.

Recibe los nuevos issues en tu correo

Un resumen breve de issues de GitHub para principiantes.