Azure-Samples / Azure-Samples/tutor
[Incident][108dev] Questions evaluation fallback: no usable graders / empty Foundry runtime config
- Dominant language
- Python
- Stars
- 17
- Forks
- 8
- PR merge metrics
- No merged PRs in 30d
Description
## Summary
Questions evaluation is falling back to baseline because there are no usable graders in the questions flow.
## Evidence
- `GET /api/questions/graders` returns `[]` (200)
- `GET /api/questions/assemblies` returns only `asm-essay-*` entries with `agents: []`
- Repro call to `POST /api/questions/grader/interaction` returns baseline with:
- `Assembly 'asm-essay-fund1-001' has no graders`
- Runtime env on `tutor-questions-108dev`:
- `PROJECT_ENDPOINT` = empty
- `MODEL_DEPLOYMENT_NAME` = empty
- Foundry specialist check:
- Foundry project exists (`.../tutor-108dev-ai-project`)
- model deployments exist (`gpt-5`, `gpt-5-nano`)
- project currently has zero agents
## Impact
- Users cannot get real multi-grader evaluation on Questions; only fallback baseline feedback appears.
## Proposed remediation
1. Restore/set AI env vars for `questions` (`PROJECT_ENDPOINT`, `MODEL_DEPLOYMENT_NAME`).
2. Re-seed question assemblies/graders so `asm-quest-*` exists with non-empty `agents`.
3. Add deploy guardrails to fail when AI env vars are empty.
4. Keep #145 for cross-domain assembly contamination tracking.
Contributor guide
Assessment
This issue has not been assessed yet.