Azure-Samples / Azure-Samples/ai-rag-chat-evaluator

Evaluation Script Fails Due to Invalid JSON Response from Authenticated Endpoint

Open
#97 10 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
335
Forks
113
Avg merge
3d 15h
Merged PRs (30d)
1

Description

### This issue is for a: (mark with an `x`)
```
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```

### Minimal steps to reproduce
1. Clone the `ai-rag-chat-evaluator` repository.
2. Follow the setup instructions to install dependencies and configure the environment.
3. Run the evaluation script with the provided command:
```sh
python -m scripts.evaluate --config=config.json

### Any log messages given by the failure
@myname ➜ /workspaces/ai-rag-chat-evaluator (main) $ python -m scripts evaluate --config=config.json
[17:57:29] INFO Running evaluation from config /workspaces/ai-rag-chat-evaluator/config.json
INFO Replaced results_dir in config with timestamp
INFO Using Azure OpenAI Service with Azure Developer CLI Credential
INFO Running evaluation using data from /workspaces/ai-rag-chat-evaluator/example_input/qa.jsonl
INFO Sending a test question to the target to ensure it is running...
ERROR Failed to send a test question to the target due to error:
Response from target https://MYBACKEND.azurewebsites.net/chat is not valid JSON:
Make sure that your configuration points at a chat endpoint that returns a single JSON object.
ERROR Evaluation was terminated early due to an error ⬆

### Expected/desired behavior
The evaluation script should successfully communicate with the chat endpoint, and the evaluation should proceed without errors.

### OS and Version?
> Ubuntu 20.04 LTS

### Versions
> Python: 3.10
> Scripts version: Latest from the main branch as of 17.07.2024

### Mention any other details that might be useful
The target_url in my config.json points to https://MYBACKEND.azurewebsites.net/chat.
The chat endpoint should return a single JSON object but seems not to be in the expected format.
The application is deployed as per the instructions in the repository documentation.
The app at https://MYBACKEND.azurewebsites.net has enabled authentication which might be affecting the evaluation script.

Contributor guide

Open the contributing guide

Research direction

Start in the scripts.evaluate entry point and inspect how config.json target_url responses are requested and parsed. Reproduce the command against the authenticated Azure endpoint, compare the returned response with the expected single JSON object, and confirm done when the evaluation proceeds successfully or the incompatibility is clearly identified.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
backend, cli, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.