Improve Visibility & Actionability of Nightly Build Failures

User Story

Problem

Our nightly build failures are not being consistently noticed or acted upon by the relevant stakeholders. Currently, the primary indicator of a failed nightly build is a build badge on the repository. This passive notification method means critical build failures often go unaddressed for prolonged periods. Crucially, developers often only discover these failures when they directly block their work, forcing them to manually search for someone who can help investigate and resolve the issue. This impacts our ability to quickly identify and resolve regressions, ultimately slowing down our development cycle and potentially impacting release quality.

Current State

• The only indication of a nightly build failure is a build badge on the repository.
• There are no automated notifications sent to Slack channels (e.g., #guild-dev-pipeline, #log-core-cicd, #log-build-status) when a nightly build fails.
• Developers find out about failures reactively, typically when they encounter a build block.

Desired Outcome

We need to ensure that nightly build failures are promptly noticed and investigated, leading to faster resolution. This means establishing active notification mechanisms and improving the visibility and actionability of these critical alerts, preventing developers from being blocked and having to manually chase down solutions.

Proposed Solutions / Ideas for Discussion

Here are some initial thoughts on how we can improve this. We should discuss these and other potential solutions as a guild:

1. Implement Automated Slack Notifications:

   • Configure our CI/CD pipeline (e.g., Jenkins, GitLab CI, etc.) to automatically send notifications to Slack channels when a nightly build fails.
   • Prioritize sending these to highly visible channels such as #log-core-cicd and #log-build-status, which are intended for status updates and likely have more active monitoring.
   • Consider also sending a summary to #guild-dev-pipeline as the primary group responsible for the pipeline's health.

2. Automated GitHub Issue Creation:

   • Investigate integrating our CI/CD pipeline with GitHub to automatically create a new GitHub issue when a nightly build fails.
   • This issue could include key details like:
     • Build URL
     • Link to logs
     • Error summary
     • Timestamp of failure
   • Automating issue creation would provide a persistent record of the failure and allow for easier tracking, assignment, and discussion, reducing the need for manual searching.

3. Enhanced Slack Notification Content (if chosen as the primary method):

   • If we primarily rely on Slack notifications, ensure they are attention-grabbing. This could include:
     • Using a prominent Slack notification format (e.g., danger attachment color, specific emojis).
     • Including direct links to build logs and relevant failure details within the Slack message.
     • Potentially @mentioning a rotating "on-call" guild member or a specific team if the failure is related to a known area.

4. Review Guild Communication Strategy:

   • As a guild, let's discuss our current communication strategy for critical pipeline events. Are there other channels or mechanisms we should be leveraging to ensure these failures are seen and acted upon proactively?

Acceptance Criteria (for eventual solution)

• Nightly build failures trigger a timely, active notification.
• The responsible party (guild member or team) is aware of the failure and begins investigation within ASAP after occurrence (likely first thing next business day), before it significantly blocks multiple developers.
• Developers are rarely, if ever, blocked by a nightly build failure that has not already been identified and acknowledged.
• There is a clear and traceable record of nightly build failures and their resolution.

Start by reviewing the existing nightly CI/CD configuration, such as Jenkins or GitLab CI, and how failures are currently surfaced through the repository build badge. Compare the proposed Slack notifications and GitHub issue workflow against the acceptance criteria, then establish which approach the guild will use. Done means failures trigger timely notifications and leave a clear, traceable record.