feat(ci): automated proto drift detection and SDK sync notifications
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 8.7k
- Forks
- 1.3k
- Avg merge
- 2d 11h
- Merged PRs (30d)
- 253
Description
User Story
As a contributor changing proto definitions, I want to know immediately if my changes affect any SDK, so that I can regenerate the stubs myself or trust that SDK maintainers will be notified automatically. Contributors shouldn't need to understand every SDK's internals to make a proto change.
As an SDK maintainer, I want to be notified when proto changes introduce fields or RPCs that my SDK's converter layer doesn't yet expose, enabling planned and prioritized SDK update work without manually tracking every proto change.
Problem Statement
OpenShell maintains four SDKs (Go, TypeScript, Python, Rust) across different teams. Proto file changes have varying impacts per SDK:
Go is unique in committing generated proto stubs to the repository. Go's source-level dependency model (go get clones and compiles sources directly) necessitates this. When protos change, committed stubs must be regenerated and re-committed. The existing go:proto:check CI task verifies stub consistency but is a simple pass/fail gate without notifications.
TypeScript, Python, and Rust all generate stubs at build time and gitignore them. TypeScript uses buf generate (output to src/gen/, gitignored). Python runs grpc_tools.protoc via a python:proto task before builds. Rust uses tonic-prost-build in a build.rs script at compile time. Stub drift is structurally impossible for these three SDKs.
All four SDKs share a deeper problem: each has a hand-written domain type layer (converters, typed wrappers, From impls) mapping proto types into idiomatic language constructs. A new proto field compiles and tests pass, but the field is silently absent from the SDK's public API. SDK consumers discover missing fields only through trial and error. This "converter coverage gap" grows silently as proto definitions evolve.
| SDK | Stubs Committed? | Generation Tool | Domain Type Layer | Stub Drift Risk |
|---|---|---|---|---|
| Go | Yes (source-level deps require it) | buf generate |
types/ + converter/ |
Yes |
| TypeScript | No (src/gen/ gitignored) |
buf generate |
Curated API + raw escape hatch |
No |
| Python | No (_proto/ gitignored) |
grpc_tools.protoc |
sandbox.py dataclass wrappers |
No |
| Rust | No (compile-time build.rs) |
tonic-prost-build |
types.rs with From impls |
No |
Impact / Why This Matters
Current behavior: Proto changes land without any notification to SDK maintainers. For Go, contributors must know to regenerate and commit stubs. For all four SDKs, new proto fields are silently ignored by the domain type layer.
Current workaround: SDK maintainers manually monitor proto changes by watching the commit log. This is error-prone and doesn't scale.
Why the workaround is insufficient: As the contributor base grows, proto changes come from many people who may not be aware of SDK implications. SDK consumers depend on proto fields being exposed through the SDK. Silent gaps create downstream surprises.
Proposed Design
Stub drift detection + issue lifecycle
Scope: Go SDK (the only SDK committing generated stubs). Infrastructure is designed to be extensible if other SDKs' build models change.
- Per-SDK mise tasks (
go:proto:drift) that regenerate stubs in a temp directory and compare against committed files, outputting structured JSON - A PR-triggered GitHub workflow annotating PRs with drift warnings (non-blocking, informational). Contributors see messages like: "Your proto change affects SDK stubs. The Go SDK stubs are 3 files behind."
- A daily cron workflow that:
- Runs drift detection for each SDK
- If drift is found, runs a build verification pipeline (regenerate, build, test)
- Creates or updates a GitHub issue per SDK (deduplicated by
sdk:{sdk}:synclabel, matching the repo's existingarea:sdk:{language}convention) - Auto-closes issues when drift resolves
- A Python script for issue body generation and lifecycle management via
gh
This makes contributions transparent without blocking anyone. Contributors see warnings; SDK maintainers get actionable issues with fix commands.
Extensibility
The system should use a config-driven approach where adding a new SDK requires only:
- A drift detection mise task for that language
- A config entry with display name, source directories, and build/test tasks
This pattern scales to all current and future SDKs.
Future extensions
Converter coverage analysis — compare proto message fields against what each SDK's converter/domain-type layer actually maps, reporting fields present in proto but absent from SDK type mappings. This would detect the deeper "business logic gap" that stub comparison cannot catch. Scope would cover all four SDKs.
Acceptance Criteria
- PR-triggered workflow annotates proto-touching PRs with SDK drift warnings (Go; extensible)
- Daily cron creates issues when drift is detected, with fix commands and affected file lists
- Issues auto-close when drift resolves
- Currently covers Go SDK stub drift (only SDK that commits stubs); matrix design supports future SDKs
- Adding a new SDK requires only a mise task + config entry
- Unit tests cover issue body generation and lifecycle management
Alternatives Considered
Fail CI on proto drift: Rejected. This would block proto contributors until every affected SDK is updated. Contributors should be able to land proto changes without understanding SDK internals. SDK maintenance is the SDK maintainer's responsibility, triggered by notifications.
Wiki dashboard: Considered and deferred. Wiki pages have low discoverability. PR annotations and auto-filed issues are more visible to those who need to act.
Integrate into existing branch-checks.yml: Worth discussing. A "Proto Drift" row alongside existing checks increases visibility. A separate workflow is easier to iterate on initially and can be promoted later.
Extend stub drift to TypeScript/Python/Rust: Not needed. All three generate stubs at build time and gitignore them, making stub drift structurally impossible. Go is the only SDK where stubs are committed. All four SDKs would benefit from the converter coverage analysis extension.
cc @maxdubrinsky @drew @mrunalp @Gkrumbach07 @rhuss
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the existing go:proto:check task and branch-checks.yml, then inspect the SDK task conventions and the proposed Python issue-management script. Define the workflow, drift task, config, and lifecycle tests around the listed acceptance criteria. Done means PR warnings, daily deduplicated issues, auto-closing, and extensible SDK configuration are covered.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions, go, python, rust, typescript
- Domain
- ci-cd, devops, tooling
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Active
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100