docling-project / docling-project/docling

Actually Using OpenAI for Annotation

Open
#1,232 1 comment 0 reactions 0 assignees View on GitHub
question triage/close-stale
Dominant language
Python
Stars
66.4k
Forks
4.8k
Avg merge
2d 21h
Merged PRs (30d)
84

Description

### Question
I have seen that it should be possible to interact with the OpenAI image API for image annotation.
e.g. in this PR: https://github.com/docling-project/docling/issues/192
Now, I have been wondering, how excatly this should be done?

It seems that OpenAI expects:
```json
{
"model": "gpt-4-vision-preview",
"messages": [
{
"role": "user",
"content": [
{ "type": "text", "text": "Describe the image in three sentences." },
{ "type": "image_url", "image_url": { "url": "data:image/png;base64,..."} }
]
}
],
"max_tokens": 200
}
```

But PictureDescriptionApiOptions gives:
```json
{
"model": "gpt-4-vision-preview",
"prompt": "...",
"image": ""
}
```

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.