Azure-Samples / Azure-Samples/ai-rag-chat-evaluator
Evaluation Script Fails Due to Invalid JSON Response from Authenticated Endpoint
- Dominant language
- Python
- Stars
- 335
- Forks
- 113
- Avg merge
- 3d 15h
- Merged PRs (30d)
- 1
Description
### This issue is for a: (mark with an `x`)
```
- [x] bug report -> please search issues before submitting
- [ ] feature request
- [ ] documentation issue or request
- [ ] regression (a behavior that used to work and stopped in a new release)
```
### Minimal steps to reproduce
1. Clone the `ai-rag-chat-evaluator` repository.
2. Follow the setup instructions to install dependencies and configure the environment.
3. Run the evaluation script with the provided command:
```sh
python -m scripts.evaluate --config=config.json
### Any log messages given by the failure
@myname ➜ /workspaces/ai-rag-chat-evaluator (main) $ python -m scripts evaluate --config=config.json
[17:57:29] INFO Running evaluation from config /workspaces/ai-rag-chat-evaluator/config.json
INFO Replaced results_dir in config with timestamp
INFO Using Azure OpenAI Service with Azure Developer CLI Credential
INFO Running evaluation using data from /workspaces/ai-rag-chat-evaluator/example_input/qa.jsonl
INFO Sending a test question to the target to ensure it is running...
ERROR Failed to send a test question to the target due to error:
Response from target https://MYBACKEND.azurewebsites.net/chat is not valid JSON:
Make sure that your configuration points at a chat endpoint that returns a single JSON object.
ERROR Evaluation was terminated early due to an error ⬆
### Expected/desired behavior
The evaluation script should successfully communicate with the chat endpoint, and the evaluation should proceed without errors.
### OS and Version?
> Ubuntu 20.04 LTS
### Versions
> Python: 3.10
> Scripts version: Latest from the main branch as of 17.07.2024
### Mention any other details that might be useful
The target_url in my config.json points to https://MYBACKEND.azurewebsites.net/chat.
The chat endpoint should return a single JSON object but seems not to be in the expected format.
The application is deployed as per the instructions in the repository documentation.
The app at https://MYBACKEND.azurewebsites.net has enabled authentication which might be affecting the evaluation script.
Contributor guide
Research direction
Start in the scripts.evaluate entry point and inspect how config.json target_url responses are requested and parsed. Reproduce the command against the authenticated Azure endpoint, compare the returned response with the expected single JSON object, and confirm done when the evaluation proceeds successfully or the incompatibility is clearly identified.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- azure, python
- Domain
- backend, cli, testing-qa
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100