Add tool to fetch latest Azure SDK versions per language
- Dominant language
- C#
- Stars
- 3.7k
- Forks
- 624
- Avg merge
- 2d 20h
- Merged PRs (30d)
- 220
Description
## Problem
LLMs consistently generate code using **outdated Azure SDK package versions** across multiple languages (Java, Python, JavaScript), regardless of configuration (baseline, skills, or Azure MCP).
### Evidence from hyoka Evaluation
From the [hyoka evaluation results](https://gist.github.com/samvaity/42ac2636028e25acb9f9c2900b960cac#4-universal-weaknesses-across-all-configs):
> **Latest Package Versions**: 9/9 ❌
> All configs generate outdated SDK versions (e.g., `azure-storage-blob 12.28.0` vs latest `12.30.0`)
This pattern appears across:
- ❌ Baseline (no tools, no skills)
- ❌ Baseline with azure-sdk-java plugin skills
- ❌ Azure MCP server
### Root Cause
LLMs are trained months prior to current SDK releases. When generating code, they reference SDK versions that were "latest" during their training cutoff, which are now outdated.
**This affects:**
- Users missing critical hotfixes
- Users missing new features and performance improvements
- Potential security vulnerabilities from outdated packages
- Overall developer experience (outdated examples in generated code)
## Proposed Solutions
### Option 1: MCP Tool to Fetch Latest SDK Versions
Add a tool to the Azure MCP server that can fetch the latest SDK version for a given service and language.
**Usage from skills:**
```markdown
# Use the Azure SDK Version tool to get the latest package
# Query: get_latest_sdk_version(service="storage", language="java")
# This returns: com.azure:azure-storage-blob:12.30.0
```
**Pros:**
- Centralized, reusable across all skills
- Can be called explicitly from skill content
- Works even without skills (baseline + Azure MCP)
**Cons:**
- Adds another tool to the MCP server
- LLM may not always choose to call it
- Needs maintenance as package managers/registries change
### Option 2: Python Code in Skills
Embed Python code directly in skill content that processes SDK release information from GitHub.
**Implementation:**
- Process `CHANGELOG.md` or release notes from Azure SDK repos
- Generate a curated version list file in a clean format
- Skills reference this file for latest versions
**Pros:**
- No new MCP tool needed
- Skills team owns the solution
- Can include additional metadata (release notes, breaking changes)
**Cons:**
- Rate limiting when hitting GitHub API
- Duplicate logic across language-specific skills
- Skills need to be regenerated when versions update
### Option 3: Enhanced Skill Content with Version Guidance
Update skills to reference SDK README files or documentation pages that list current versions.
**Example:**
```markdown
# For latest versions, check:
# - Python: https://github.com/Azure/azure-sdk-for-python/blob/main/sdk/storage/azure-storage-blob/README.md
# - Java: https://github.com/Azure/azure-sdk-for-java/blob/main/sdk/storage/azure-storage-blob/README.md
# - .NET: https://github.com/Azure/azure-sdk-for-net/blob/main/sdk/storage/Azure.Storage.Blobs/README.md
```
**Pros:**
- No new infrastructure needed
- Points LLM to authoritative sources
- Works with existing web_fetch capabilities
**Cons:**
- LLM may not follow the guidance
- Adds latency to fetch README content
- No guarantee the LLM will parse version correctly
## Recommendation
**Start with Option 1 (MCP Tool)** for immediate impact, with Option 3 (Skill Content) as a complementary approach?
## Open Questions
1. **Skill Integration**: Should skills explicitly instruct to call this tool, or rely on tool description?
2. **Caching**: How often should version data be refreshed? (Daily? Weekly?)
3. **Scope**: Should this handle all Azure packages or start with a subset (e.g., top 10 services)
Contributor guide
Assessment
This issue has not been assessed yet.