AnswerDotAI / AnswerDotAI/claudette
Is there a way to have a tool return image bytes?
- Dominant language
- Jupyter Notebook
- Stars
- 316
- Forks
- 43
- PR merge metrics
- No merged PRs in 30d
Description
First, thanks for claudette. It's very elegant.
`toolslm` doesn't appear to map the "bytes" type to anything, which I guess makes sense.
However, that means I'm having a hard time using the toolloop if I include my image returning function.
```python
def get_image_of_puppy() -> bytes:
"Returns an image of a puppy"
image: Path = Path("puppy.jpg")
return image.read_bytes()
tools = [get_image_of_puppy]
chat = Chat(model, tools=tools)
r = chat.toolloop("Describe the puppy image")
print(contents(r))
```
Error:
```
Traceback (most recent call last):
File "/Users/taytay/projects/llm-browser-driver/src/llm_browser_driver/playground/./test_claude.py", line 28, in
r = chat.toolloop("Describe the puppy image")
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/toolloop.py", line 23, in toolloop
r = self(pr, **kwargs)
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/core.py", line 200, in __call__
if self.tools: kw['tools'] = [get_schema(o) for o in self.tools]
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/core.py", line 200, in
if self.tools: kw['tools'] = [get_schema(o) for o in self.tools]
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/toolslm/funccall.py", line 43, in get_schema
if ret.anno is not empty: desc += f'\n\nReturns:\n- type: {_types(ret.anno)[0]}'
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/toolslm/funccall.py", line 20, in _types
else: return tmap[t], None
KeyError:
```
Which of course comes from:
```
# %% ../01_funccall.ipynb 11
def _types(t:type)->tuple[str,Optional[str]]:
"Tuple of json schema type name and (if appropriate) array item name."
if t is empty: raise TypeError('Missing type')
tmap = {int:"integer", float:"number", str:"string", bool:"boolean", list:"array", dict:"object"}
if getattr(t, '__origin__', None) in (list,tuple): return "array", tmap.get(t.__args__[0], "object")
else: return tmap[t], None
```
I tried returning a dict of the image type, but claude complains there were too many input tokens. I need to get the image to get sent as a group of bytes back into the chat so its machinery kicks in and converts it to an image that Claude understands:
```python
def img_msg(data:bytes)->dict:
"Convert image `data` into an encoded `dict`"
img = base64.b64encode(data).decode("utf-8")
mtype = mimetypes.types_map['.'+imghdr.what(None, h=data)]
r = dict(type="base64", media_type=mtype, data=img)
return {"type": "image", "source": r}
```
I could probably hack it together, but I keep feeling like I'm missing something obvious?
Contributor guide
No contributing guide indexed for this repository
Assessment
This issue has not been assessed yet.