microsoft / microsoft/promptflow

[BUG] Content Filter errors originating from LLM steps contain a mix of JSON and text representations of Python objects

Open
#1,735 3 comments 0 reactions 3 assignees View on GitHub

@zzn2 is already working on this.

Since Jan 15, 2024.

feature long-term
Dominant language
Python
Stars
11.2k
Forks
1.1k
PR merge metrics
No merged PRs in 30d

Description

Describe the bug
When an LLM step in a PromptFlow run generates an error, it produces an error message format that is very difficult to consume with code. For example:

{ "error": { "code": "UserError", "message": "OpenAI API hits BadRequestError: Error code: 400 - {'error': {'message': \"The response was filtered due to the prompt triggering Azure OpenAI's content management policy. Please modify your prompt and retry. To learn more about our content filtering policies please read our documentation: https://go.microsoft.com/fwlink/?linkid=2198766\", 'type': None, 'param': 'prompt', 'code': 'content_filter', 'status': 400, 'innererror': {'code': 'ResponsibleAIPolicyViolation', 'content_filter_result': {'hate': {'filtered': False, 'severity': 'safe'}, 'self_harm': {'filtered': True, 'severity': 'medium'}, 'sexual': {'filtered': False, 'severity': 'safe'}, 'violence': {'filtered': False, 'severity': 'safe'}}}}} [Error reference: https://platform.openai.com/docs/guides/error-codes/api-errors]" } }

In this specific scenario, I'd want to have an error handler that could identify that the error was related to content filtering, and be able to do different things depending on which filter was hit, and to what degree. This is difficult with the current error, however, because it's a JSON object containing a string object which itself seems to contain a string representation of a Python object.

How To Reproduce the bug

  1. Set up a basic Prompt Flow with an LLM step, and the default Content Filter policy
  2. Send input to the flow that triggers the content filter policy

Expected behavior
The PromptFlow should return an error that is fully JSON formatted, not a mix of JSON and a text representation of a Python object

Running Information(please complete the following information):

  • Promptflow Package Version: Using the hosted preview on ai.azure.com

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.