Project: Reasoning Agents - MS CertiMentor

Track

Reasoning Agents (Azure AI Foundry)

Project Name

MS CertiMentor

GitHub Username

@yaliagap

Repository URL

https://github.com/yaliagap/MS-CertiMentor-Reasoning-Agent

Project Description

MS-CertiMentor: AI-Powered Certification Preparation System

What It Does

MS-CertiMentor is an intelligent multi-agent system that guides students through their Microsoft certification journey. Using six specialized AI agents working in orchestrated workflows, it provides personalized study plans, adaptive assessments, and actionable exam readiness recommendations.

Problem It Solves

Students preparing for Microsoft certifications face common challenges: finding relevant learning materials, creating realistic study schedules, staying motivated, and knowing when they're truly ready to book their exam. MS-CertiMentor addresses these pain points by providing a comprehensive, AI-guided preparation experience that adapts to each student's pace and performance.

Key Features

Multi-Agent Architecture: Six specialized agents collaborate sequentially:

• Learning Path Curator: Discovers relevant Microsoft Learn content
• Study Plan Generator: Creates realistic timelines with daily sessions and milestones
• Engagement Agent: Schedules motivational reminders to maintain consistency
• Assessment Agent: Generates certification-style practice quizzes
• Assessment Evaluator: Provides detailed educational feedback on performance
• Exam Plan Agent: Recommends certification exams with readiness assessment

Advanced Reasoning Patterns: Implements planner-executor, iterative refinement, and human-in-the-loop checkpoints for optimal decision-making.

Enterprise Observability: Full Azure Application Insights integration with OpenTelemetry for tracking agent interactions, performance metrics, and workflow telemetry.

Adaptive Learning Loop: Failed assessments trigger focused re-study with targeted recommendations, ensuring students are genuinely prepared before booking expensive certification exams.

Built with Microsoft Agent Framework and Azure OpenAI Service.

Demo Video or Screenshots

Screenshots: https://github.com/yaliagap/MS-CertiMentor-Reasoning-Agent/tree/main/screenshoots

Primary Programming Language

Python

Key Technologies Used

Core Framework & AI

• Microsoft Agent Framework (v1.0.0rc2) - Multi-agent orchestration and workflow management
• Azure OpenAI Service - GPT-4o for intelligent agent reasoning and responses
• Python 3.13 - Primary development language

Data & Validation

• Pydantic v2 - Structured data models and validation for agent inputs/outputs
• asyncio - Asynchronous workflow execution and agent coordination

Observability & Monitoring

• Azure Application Insights - Enterprise-grade telemetry and monitoring
• OpenTelemetry - Distributed tracing for multi-agent workflows
• azure-monitor-opentelemetry - Seamless Azure integration

Agent Framework Extensions

• agent-framework-azure-ai - Azure AI integration layer
• agent-framework-core - Core agent primitives and abstractions

Development Tools

• python-dotenv - Environment configuration management
• typing-extensions - Enhanced type hints for better code safety

Submission Type

Individual

Team Members

No response

Submission Requirements

• My project meets the track-specific challenge requirements
• My repository includes a comprehensive README.md with setup instructions
• My code does not contain hardcoded API keys or secrets
• I have included demo materials (video or screenshots)
• My project is my own work with proper attribution for any third-party code
• I agree to the Code of Conduct
• I have read and agree to the Disclaimer
• My submission does NOT contain any confidential, proprietary, or sensitive information
• I confirm I have the rights to submit this content and grant the necessary licenses

Quick Setup Summary

1. Clone the repository

   git clone <repository-url>
   cd MS-CertiMentor-Reasoning-Agent
   

2. Install dependencies

   pip install -r requirements.txt
   

3. Configure environment variables

   cp .env.example .env
   # Edit .env with your Azure OpenAI credentials
   

4. Run the system

   python main.py
   

5. Follow the interactive prompts to enter your study topic, experience level, and schedule preferences.

The system will guide you through the complete certification preparation workflow, from learning path curation to exam readiness assessment.

For full documentation, deployment options, and troubleshooting, see README.md.

Technical Highlights

Architecture & Design Decisions

Intelligent Agent Separation: Split assessment into two specialized agents (Quiz Generator + Educational Evaluator) rather than one monolithic agent. This separation allows the quiz generator to focus purely on question quality while the evaluator provides rich, pedagogical feedback without conflating concerns.

Adaptive Learning Loop with Guardrails: Implemented iterative refinement pattern with a 3-attempt maximum. Failed assessments don't just loop endlessly—the system provides targeted feedback and forces re-study of weak domains before retry, simulating real learning progression.

Human-in-the-Loop at Critical Junctures: Strategic checkpoint before assessment ensures students don't waste AI resources on assessments they know they're unprepared for, balancing automation with human judgment.

Technical Implementation

Comprehensive Observability: Full OpenTelemetry instrumentation across all 6 agents. Every agent interaction, tool call, and decision is traced through Azure Application Insights, enabling deep debugging and performance analysis of multi-agent reasoning chains.

Type-Safe Data Models: Leveraged Pydantic v2 for complex nested data structures (exam plans with domain breakdowns, assessment feedback with per-question analysis). Field validators ensure data integrity (e.g., URLs must be microsoft.com/pearsonvue.com, scores 0-100, proper enum values).

Resilient JSON Handling: Implemented fallback parsing with detailed error context when structured outputs fail, including debug file generation and position-aware error messages for rapid troubleshooting.

Temperature Tuning by Role: Each agent has purpose-driven temperature settings (0.2 for objective quizzes, 0.6 for creative motivation, 0.3 for factual guidance), optimizing reasoning quality per task.

Challenges & Learnings

Challenge 1: Structured Output Reliability

Problem: Azure OpenAI's response_format with Pydantic models didn't always provide native structured outputs, forcing fallback to text-based JSON parsing. This led to intermittent JSON parsing errors when agents generated malformed JSON.

Learning: Always implement robust fallback strategies for AI outputs. We added comprehensive error handling with context-aware debugging, position tracking for parse failures, and automatic debug file generation. When relying on AI-generated structured data, validate early and provide detailed error context.

Challenge 2: Pydantic Model Design for AI Outputs

Problem: Initially used int for score fields, but agents naturally calculated percentages as decimals (66.7%, 33.3%). This caused validation errors despite semantically correct outputs.

Learning: Design data models to match how AI models naturally represent information, not just what seems "correct" from a programming perspective. Float scores are more natural for AI reasoning about percentages.

Challenge 3: Multi-Agent State Management

Problem: Coordinating state flow between 6 sequential agents while maintaining type safety and ensuring each agent received exactly the data it needed.

Learning: Strongly-typed state dictionaries (TypedDict) combined with Pydantic models provided the perfect balance—type safety during development with flexibility at runtime. Clear interface contracts between agents prevented coupling.

Challenge 4: Balancing Automation vs Control

Problem: Determining when to automate decisions versus requiring human approval. Too much automation risks wasting resources; too many checkpoints frustrate users.

Learning: Strategic human-in-the-loop placement at high-stakes moments (before taking assessment) provided the best balance. Let AI handle routine decisions, but gate irreversible or resource-intensive actions.

Most Valuable Insight

Observability isn't optional for multi-agent systems—it's essential. OpenTelemetry traces revealed exactly where agent reasoning diverged from expectations, turning debugging from guesswork into data-driven analysis.

Contact Information

aliagapalomino@gmail.com

Country/Region

Peru

Start by reading README.md and requirements.txt in the linked MS-CertiMentor-Reasoning-Agent repository, then follow the .env.example setup and run main.py. The submission describes an existing multi-agent certification system rather than a specific change, so completion would require defining a separate implementation scope before work can begin.