Azure / Azure/typespec-azure

Enable SDK Smoke Tests to Validate Generated Code Changes

Open
#4,720 2 comments 0 reactions 1 assignee Claimed by @msyyc View on GitHub
emitter:client:all
Dominant language
TypeScript
Stars
27
Forks
90
Avg merge
1d 22h
Merged PRs (30d)
156

Description

### Background
Today, most validation focuses on payload correctness through Spector tests. While Spector is valuable for validating API behaviors and generated payloads, it does not provide sufficient coverage for validating the generated SDK code when upstream changes occur in TypeSpec, TCGC, emitters, or related code generation components.
As code generation evolves, seemingly small upstream changes can introduce unintended SDK code differences that are difficult to detect until later in the development cycle.

As we all emitters into one repo, here are a couple of challenges:

- Emitter developers have limited visibility into how changes affect generated SDK outputs.
- Impact analysis across TypeSpec, TCGC, emitters, and shared tooling is slow and inconsistent.
- AI-assisted development and automated code changes lack fast feedback signals to validate generated SDK quality.

### Actions:
- Create a lightweight SDK smoke test framework focused on generated code validation.
- Select a small set of representative services (~5 services) covering major generation scenarios.
- Automatically regenerate SDKs when TypeSpec, TCGC, emitter, or shared code generation changes occur.
- Extend Contoso service to cover common generation scenarios and use it as a primary validation service.
- Detect and surface meaningful SDK code differences and integrate smoke tests into CI pipelines and developer workflows.

Contributor guide

Open the contributing guide

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.