Copilot coding agent breaks commit syncs

👋🏻 Hello from rubygems.org

We recently had our first PR generated from the new Copilot coding agent, which, when merged, caused our Shipit Stacks to immediately break and be marked as inaccessible from:

https://github.com/Shopify/shipit-engine/blob/af872321d9bc197f4c46401874e66d3c428e90c0/app/jobs/shipit/github_sync_job.rb#L55-L61

The GitHub API makes it seem like the GitHub Copilot user is a real author, but it seems as though the API to get the Copilot user will just return 404.

$ gh api repos/rubygems/rubygems.org/pulls/5929 | jq .user
{
  "login": "Copilot",
  "id": 198982749,
  "node_id": "BOT_kgDOC9w8XQ",
  "avatar_url": "https://avatars.githubusercontent.com/in/1143301?v=4",
  "gravatar_id": "",
  "url": "https://api.github.com/users/Copilot",
  "html_url": "https://github.com/apps/copilot-swe-agent",
  "followers_url": "https://api.github.com/users/Copilot/followers",
  "following_url": "https://api.github.com/users/Copilot/following{/other_user}",
  "gists_url": "https://api.github.com/users/Copilot/gists{/gist_id}",
  "starred_url": "https://api.github.com/users/Copilot/starred{/owner}{/repo}",
  "subscriptions_url": "https://api.github.com/users/Copilot/subscriptions",
  "organizations_url": "https://api.github.com/users/Copilot/orgs",
  "repos_url": "https://api.github.com/users/Copilot/repos",
  "events_url": "https://api.github.com/users/Copilot/events{/privacy}",
  "received_events_url": "https://api.github.com/users/Copilot/received_events",
  "type": "Bot",
  "user_view_type": "public",
  "site_admin": false
}

❯ gh api users/Copilot
{
  "message": "Not Found",
  "documentation_url": "https://docs.github.com/rest",
  "status": "404"
}
gh: Not Found (HTTP 404)

I'm not particularly sure what we could do to handle this particular scenario, and we've since disabled the Copilot agent on rubygems.org. I figured the best first step would be to get your reaction and see what happens from there.

Read app/jobs/shipit/github_sync_job.rb around lines 55-61, then use the linked rubygems.org pull request and the shown gh api users/Copilot response to understand the failure. Determine how the sync handles an author lookup returning 404, and define completion as preventing this Copilot-authored merge from marking the Shipit Stack inaccessible, with regression coverage.