Auto-detect breaking changes to ensure migration guides exist
- Dominant language
- Rust
- Stars
- 48.2k
- Forks
- 4.8k
- Avg merge
- 3d 16h
- Merged PRs (30d)
- 171
Description
We chatted on Mastodon about breaking changes: https://mastodon.gamedev.place/@alice_i_cecile/111960897354647011
I believe the key points were:
- It's very important that each breaking change comes with a migration guide for users.
- Currently, pull requests are manually triaged for breaking changes. PRs with breaking changes are labeled `C-Breaking-Change`, and there's a CI job that ensures the existence of a migration guide in the PR.
- There are situations where an additional way of detecting breaking changes may be helpful. For example, if a complex PR contains multiple breaking changes, they should all come with instructions in the migration guide. This is also done via human review.
- Research shows that despite our best efforts, we humans miss things quite a lot: more than 1 in 6 of the top 1000 Rust crates has accidentally shipped a breaking change at least once, and over 3% of all releases we tested had breakage that wasn't supposed to be there: https://predr.ag/blog/semver-violations-are-common-better-tooling-is-the-answer/
If you're interested, I believe it to be possible to use `cargo-semver-checks` to independently analyze a PR's contents and produce a list of the breaking changes it found. This list may be incomplete — we still have many not-yet-implemented lints — but together with the reviewer's judgment will likely catch many instances of breaking changes and missing migration guides that otherwise may have slipped through uncaught.
Contributor guide
Research direction
Start by reviewing the existing CI job that checks for migration guides on pull requests, then investigate how cargo-semver-checks can analyze PR contents and report breaking changes. The work is done when the automated analysis is integrated with the existing review flow and helps identify breaking changes that lack migration guidance.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- ci-cd, devtools, documentation
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100