Azure-Samples / Azure-Samples/tutor

[Incident][108dev] Questions evaluation fallback: no usable graders / empty Foundry runtime config

Open
#146 4 comments 0 reactions 0 assignees View on GitHub
Dominant language
Python
Stars
17
Forks
8
PR merge metrics
No merged PRs in 30d

Description

## Summary
Questions evaluation is falling back to baseline because there are no usable graders in the questions flow.

## Evidence
- `GET /api/questions/graders` returns `[]` (200)
- `GET /api/questions/assemblies` returns only `asm-essay-*` entries with `agents: []`
- Repro call to `POST /api/questions/grader/interaction` returns baseline with:
- `Assembly 'asm-essay-fund1-001' has no graders`
- Runtime env on `tutor-questions-108dev`:
- `PROJECT_ENDPOINT` = empty
- `MODEL_DEPLOYMENT_NAME` = empty
- Foundry specialist check:
- Foundry project exists (`.../tutor-108dev-ai-project`)
- model deployments exist (`gpt-5`, `gpt-5-nano`)
- project currently has zero agents

## Impact
- Users cannot get real multi-grader evaluation on Questions; only fallback baseline feedback appears.

## Proposed remediation
1. Restore/set AI env vars for `questions` (`PROJECT_ENDPOINT`, `MODEL_DEPLOYMENT_NAME`).
2. Re-seed question assemblies/graders so `asm-quest-*` exists with non-empty `agents`.
3. Add deploy guardrails to fail when AI env vars are empty.
4. Keep #145 for cross-domain assembly contamination tracking.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.