Abraxas-365 / Abraxas-365/langchain-rust

Chain does not support images

未關閉
#238 0 則留言 0 個 reaction 已指派 0 人 在 GitHub 檢視
主要語言
Rust
星號
1.3k
分支
177
PR 合併指標
30 天內沒有已合併 PR

描述

**Describe the bug**
Using chain (`ConversationalChainBuilder` or `Agent`) does not support images.

```rust
prompt_args! {
"input" => input,
}
```

does not allow to pass an `image url`.
If the `url` is passed as a string the in input, an error is thrown `OpenAI error: stream failed: Invalid status code: 400 Bad Request` probably because the image is not under the right structure ([`Image content part`](https://platform.openai.com/docs/guides/vision)).

Also
`langchain_rust::schemas::Message::messages_to_string` only serialize text content. It does not take in consideration images.
https://github.com/Abraxas-365/langchain-rust/blob/891c56f0e2f2a98eed883f906c3de6cb82770797/src/schemas/messages.rs#L149-L155

**To Reproduce**
```rust
let chain = ConversationalChainBuilder::new()
.llm(llm)
.build();

let base64_image_url = "data:image/jpeg;base64,23434234234234".to_string();
let input = format!("What is that? {}", base64_image_url);
let result = chain.invoke(prompt_args! {
"input" => input,
});
```

**Expected behavior**
A way to pass image to a chain or at minimum an Agent.

Since images are on `langchain_rust::schemas::Message` structure, I would expect them to work in every aspect of the library.

**Additional context**
This is some kind of a violation of the `Liskov Principle` & `Interface Segregation Principles`. If image are available on `langchain_rust::schemas::Message` they should be supported everywhere in the library.

貢獻指南

開啟貢獻指南

研究方向

The issue points to `langchain_rust::schemas::Message::messages_to_string` in src/schemas/messages.rs, which currently only serializes text content. Start by examining the Message struct to understand how image content is stored. Then, look at ConversationalChainBuilder and Agent to see how messages are processed and where image support needs to be added. Testing will involve creating a chain with an image URL and ensuring it conforms to OpenAI's vision API structure.

由索引模型根據 Issue 內容生成。

評估

技術堆疊
rust
領域
ai-infra-agents
Issue 類型
缺陷
難度
4/5
預估耗時
3-5 天
活躍度
停滯
描述清晰度
描述清楚
新手友好度
45/100

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。