Abraxas-365 / Abraxas-365/langchain-rust
Chain does not support images
- Lenguaje dominante
- Rust
- Estrellas
- 1.3k
- Forks
- 177
- Métricas de merge de PR
- Sin PR fusionados en 30 d
Descripción
**Describe the bug**
Using chain (`ConversationalChainBuilder` or `Agent`) does not support images.
```rust
prompt_args! {
"input" => input,
}
```
does not allow to pass an `image url`.
If the `url` is passed as a string the in input, an error is thrown `OpenAI error: stream failed: Invalid status code: 400 Bad Request` probably because the image is not under the right structure ([`Image content part`](https://platform.openai.com/docs/guides/vision)).
Also
`langchain_rust::schemas::Message::messages_to_string` only serialize text content. It does not take in consideration images.
https://github.com/Abraxas-365/langchain-rust/blob/891c56f0e2f2a98eed883f906c3de6cb82770797/src/schemas/messages.rs#L149-L155
**To Reproduce**
```rust
let chain = ConversationalChainBuilder::new()
.llm(llm)
.build();
let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
"input" => input,
});
```
**Expected behavior**
A way to pass image to a chain or at minimum an Agent.
Since images are on `langchain_rust::schemas::Message` structure, I would expect them to work in every aspect of the library.
**Additional context**
This is some kind of a violation of the `Liskov Principle` & `Interface Segregation Principles`. If image are available on `langchain_rust::schemas::Message` they should be supported everywhere in the library.
Guía de contribución
Evaluación
Este issue todavía no se ha evaluado.