Azure-Samples / Azure-Samples/get-started-with-ai-agents

pytest tests/test_red_teaming.py -s is failed

Open
#229 0 comments 0 reactions 0 assignees View on GitHub
Dominant language
Bicep
Stars
366
Forks
729
PR merge metrics
No merged PRs in 30d

Description

Hi,
While running test_red_teaming.py -s, I am facing an issue. Could you please help me?
Please find the error below.

platform linux -- Python 3.11.13, pytest-8.4.2, pluggy-1.6.0
rootdir: /workspaces/get-started-with-ai-agents
plugins: anyio-4.13.0
collected 1 item

tests/test_red_teaming.py Agent retrieved (id: agent-template-assistant:1, name: agent-template-assistant, version: 1)
[{'evaluator_name': 'builtin.prohibited_actions',
'name': 'Prohibited Actions',
'type': 'azure_ai_evaluator'}]
Red team evaluation created for red teaming: Red Team Agent Safety evaluation -1774940289
F

======================================================== FAILURES ========================================================
____________________________________________________ test_red_teaming ____________________________________________________

def test_red_teaming() -> None:

with (
DefaultAzureCredential(exclude_interactive_browser_credential=False) as credential,
AIProjectClient(endpoint=retrieve_endpoint(), credential=credential) as project_client,
project_client.get_openai_client() as client,
):

agent = retrieve_agent(project_client)

eval_group_name = "Red Team Agent Safety evaluation -" + str(int(time.time()))
eval_run_name = f"Red Team Agent Safety evaluation run for {agent.name} -" + str(int(time.time()))
data_source_config = {"type": "azure_ai_source", "scenario": "red_team"}

# Define testing criteria for red teaming.
# Explore evaluator catalog for assessments of additional risk categories.
testing_criteria = [
{
"type": "azure_ai_evaluator",
"name": "Prohibited Actions",
"evaluator_name": "builtin.prohibited_actions"
}
]
pprint(testing_criteria)

eval_object = client.evals.create(
name=eval_group_name,
data_source_config=data_source_config,
testing_criteria=testing_criteria,
)
print(f"Red team evaluation created for red teaming: {eval_group_name}")

risk_categories_for_taxonomy = [RiskCategory.PROHIBITED_ACTIONS]
target = AzureAIAgentTarget(
name=agent.name, version=agent.version, tool_descriptions=_get_tool_descriptions(agent)
)
agent_taxonomy_input = AgentTaxonomyInput(risk_categories=risk_categories_for_taxonomy, target=target)
eval_taxonomy_input = EvaluationTaxonomy(
description="Taxonomy for red teaming evaluation", taxonomy_input=agent_taxonomy_input
)
> taxonomy = project_client.evaluation_taxonomies.create(name=agent.name, body=eval_taxonomy_input)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
E AttributeError: 'AIProjectClient' object has no attribute 'evaluation_taxonomies'

tests/test_red_teaming.py:61: AttributeError
================================================ short test summary info =================================================
FAILED tests/test_red_teaming.py::test_red_teaming - AttributeError: 'AIProjectClient' object has no attribute 'evaluation_taxonomies'

Image

Contributor guide

Open the contributing guide

Research direction

Start with tests/test_red_teaming.py, especially the evaluation_taxonomies.create call at line 61, and inspect the AIProjectClient API and installed SDK versions shown in the report. Run pytest tests/test_red_teaming.py -s while investigating the missing client attribute; done means the red-teaming test completes without this AttributeError.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, python
Domain
ai, testing-qa
Issue type
Bug
Difficulty
3/5
Estimated time
1-2 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
48/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.