AnswerDotAI / AnswerDotAI/claudette

Is there a way to have a tool return image bytes?

Open
#10 3 comments 0 reactions 1 assignee Claimed by @ncoop57 View on GitHub
good first issue
Dominant language
Jupyter Notebook
Stars
316
Forks
43
PR merge metrics
No merged PRs in 30d

Description

First, thanks for claudette. It's very elegant.

`toolslm` doesn't appear to map the "bytes" type to anything, which I guess makes sense.
However, that means I'm having a hard time using the toolloop if I include my image returning function.

```python

def get_image_of_puppy() -> bytes:
"Returns an image of a puppy"
image: Path = Path("puppy.jpg")
return image.read_bytes()

tools = [get_image_of_puppy]

chat = Chat(model, tools=tools)
r = chat.toolloop("Describe the puppy image")
print(contents(r))

```

Error:
```
Traceback (most recent call last):
File "/Users/taytay/projects/llm-browser-driver/src/llm_browser_driver/playground/./test_claude.py", line 28, in
r = chat.toolloop("Describe the puppy image")
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/toolloop.py", line 23, in toolloop
r = self(pr, **kwargs)
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/core.py", line 200, in __call__
if self.tools: kw['tools'] = [get_schema(o) for o in self.tools]
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/claudette/core.py", line 200, in
if self.tools: kw['tools'] = [get_schema(o) for o in self.tools]
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/toolslm/funccall.py", line 43, in get_schema
if ret.anno is not empty: desc += f'\n\nReturns:\n- type: {_types(ret.anno)[0]}'
File "/Users/taytay/projects/llm-browser-driver/.venv/lib/python3.10/site-packages/toolslm/funccall.py", line 20, in _types
else: return tmap[t], None
KeyError:
```

Which of course comes from:
```
# %% ../01_funccall.ipynb 11
def _types(t:type)->tuple[str,Optional[str]]:
"Tuple of json schema type name and (if appropriate) array item name."
if t is empty: raise TypeError('Missing type')
tmap = {int:"integer", float:"number", str:"string", bool:"boolean", list:"array", dict:"object"}
if getattr(t, '__origin__', None) in (list,tuple): return "array", tmap.get(t.__args__[0], "object")
else: return tmap[t], None
```

I tried returning a dict of the image type, but claude complains there were too many input tokens. I need to get the image to get sent as a group of bytes back into the chat so its machinery kicks in and converts it to an image that Claude understands:

```python
def img_msg(data:bytes)->dict:
"Convert image `data` into an encoded `dict`"
img = base64.b64encode(data).decode("utf-8")
mtype = mimetypes.types_map['.'+imghdr.what(None, h=data)]
r = dict(type="base64", media_type=mtype, data=img)
return {"type": "image", "source": r}

```

I could probably hack it together, but I keep feeling like I'm missing something obvious?

Contributor guide

No contributing guide indexed for this repository

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.