Azure / Azure/azure-rest-api-specs

[Question] Azure OpenAI - Latest API version missing "image" message content for Assistant API

Open
#30,090 16 comments 6 reactions 0 assignees View on GitHub
customer-reported data-plane OpenAI question Service Attention
Dominant language
TypeSpec
Stars
3.1k
Forks
6k
Avg merge
2d 22h
Merged PRs (30d)
444

Description

### API Spec link

https://github.com/Azure/azure-rest-api-specs/tree/main/specification/cognitiveservices/data-plane/AzureOpenAI

### API Spec version

2024-07-01-preview

### Question/Query

## Message Content
When trying to add non-text messages we get an error message indicating only "text" is allowed. OpenAI has supported this for a couple months so it's surprising to see this missing still. Do we have an ETA on when this will be added?
```py
await client.beta.threads.messages.create(
thread_id=thread.id,
role="user",
content=[{'text': 'what does this say?', 'type': 'text'}, {'image_url': {'url': 'data:image/png;base64,iVBORw0KG...', 'detail': 'auto'}, 'type': 'image_url'}]
)
```
```bash
openai.BadRequestError: Error code: 400 - {'error': {'message': "Invalid message content: only messages with 'type=text' are supported currently.", 'type': 'invalid_request_error', 'param': 'content', 'code': 'invalid_type'}}
```

### Environment

_No response_

Contributor guide

Open the contributing guide

Research direction

Start with the specification/cognitiveservices/data-plane/AzureOpenAI path linked in the issue and inspect the 2024-07-01-preview definition for Assistant API message content. Compare its supported content types with the reported Python request and error; done would require a documented, accepted image message shape and confirmation that the Azure endpoint supports it.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, openapi
Domain
api, cloud
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.