feyninc / feyninc/catsu

[Feat]: Automated batching to respect quotas

未關閉
#29 0 則留言 0 個 reaction 已指派 1 人 已被 @chonknick 認領 在 GitHub 檢視
enhancement
主要語言
Rust
星號
83
分支
4
PR 合併指標
30 天內沒有已合併 PR

描述

### Problem Statement

When doing a batch request to OpenAI embedding models, the total number of tokens is limited to 300k ([reference](https://platform.openai.com/docs/api-reference/embeddings)). `catsu` properly surfaces that error response to the user.

```python
import catsu

lots_of_text = ["foo", "bar", "baz", ...] # totals >300k tokens

client = catsu.client()
response = client.embed("openai:text-embedding-small-3", lots_of_text)
```

This is fixed by manually batching. Illustratively
```python
import itertools
import catsu

lots_of_text = ["foo", "bar", "baz", ...] # totals >300k tokens

client = catsu.client()

responses = []
# batch items in groups of 500; arbitrary and could hit token limit
for batch in itertools.batched(lots_of_text, 500):
response = client.embed("openai:text-embedding-small-3", batch)
responses.append(response)
```

### Proposed Solution

It would be amazing if catsu could automatically batch inputs. This would involve tokenizing all inputs and would be expensive though. Maybe this should like in `chonkie` ?

```python
import catsu

lots_of_text = ["foo", "bar", "baz", ...] # totals >300k tokens

client = catsu.client()

responses = []
# batch items in groups of 500; arbitrary and could hit token limit
for batch in catsu.batch_inputs(lots_of_text):
response = client.embed("openai:text-embedding-small-3", batch)
responses.append(response)

# OR
responses = client.embed("openai:text-embedding-small-3", lots_of_text, batch_inputs=True)
```

There could be a sync and async option.

貢獻指南

開啟貢獻指南

評估

這個 Issue 還沒有評估資料。

把新 issue 寄到你的電子郵件信箱

精選適合新手參與的 GitHub issue 摘要。