Azure-Samples / Azure-Samples/azureai-samples

Azure Foundry Project Evaluation which is in preview mode, Not able to use Evaluation dashboard in portal

Open
#282 1 comment 0 reactions 0 assignees View on GitHub
bug
Dominant language
Jupyter Notebook
Stars
548
Forks
318
Avg merge
3d 15h
Merged PRs (30d)
1

Description

### Operating System

MacOS

### Version Information

We using the below approach for evaluation, out chat model, we read the data from table storage, and used for eval, but somehow on locally we are getting the result. but we are not getting any update on the dashboard, even though our evaluation is successful at local.

# Try different approaches for Azure integration
try:
# Approach 1: Try with project scope dictionary
project_scope = {
"subscription_id": CONFIG.get("AZURE_SUBSCRIPTION_ID"),
"resource_group_name": CONFIG.get("AZURE_RESOURCE_GROUP_NAME"),
"project_name": CONFIG.get("AZURE_AI_PROJECT_NAME"),
}

evaluation_result = evaluate(
data=data_file,
evaluation_name=evaluation_name,
evaluators=self._evaluators,
evaluator_config={
"default": {
"query": {"${data.query}"},
"response": {"${data.response}"},
"context": {"${data.context}"},
}
},
azure_ai_project=project_scope,
output_path=output_path,
)
logger.info(
"Azure AI Project integration successful with project scope"
)

except Exception as azure_error:
logger.warning(f"Azure integration failed: {azure_error}")
logger.info("Falling back to local evaluation only")

### Steps to reproduce

1. Configure the key in .env:
# Replace connection string with endpoint and project name
AZURE_AI_PROJECT_ENDPOINT=
AZURE_SUBSCRIPTION_ID=
AZURE_RESOURCE_GROUP_NAME=
AZURE_AI_PROJECT_NAME=

# Keep your existing OpenAI and storage configs
AZURE_OPENAI_ENDPOINT=
AZURE_OPENAI_DEPLOYMENT=
AZURE_OPENAI_API_VERSION=
AZURE_OPENAI_API_KEY=
AZURE_STORAGE_CONNECTION_STRING=
AZURE_STORAGE_TABLE_NAME=
EVALUATION_OUTPUT_PATH=

2. use the above code, and try to run evaluate

### Expected behavior

It should create a dashboard at Azure Portal in Evaluation section. i was able to do that in Foundry hub

### Actual behavior

it is not able to create the dasbaord

### Addition information

i am using the library:

from azure.ai.projects import AIProjectClient
from azure.ai.evaluation import (
evaluate,
GroundednessEvaluator,
CoherenceEvaluator,
RelevanceEvaluator,
)
from azure.identity import DefaultAzureCredential

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.