Proposal: Splitting CI pipelines into PR and Nightly workflows to accelerate development

Context & Problem
Currently, our Continuous Integration (CI) pipeline installs multiple external software independent of ARC, with more joining soon, and run comprehensive tests for every PR. While this ensures thorough testing, the prolonged CI times are becoming a bottleneck. Waiting for heavy environments to build and extensive test suites to finish significantly slows down the development cycle and delays crucial feedback on Pull Requests.

Proposed Solution
To expedite dev time without sacrificing code reliability, I propose we pivot to a two-tier CI strategy:

1. PR-Level CI (Fast Feedback)

• Trigger: Runs on every Pull Request and commit.
• Scope: Focuses on verifying core logic by running unit tests and mocking the heavy external software interfaces.
• Goal: Provide rapid, actionable feedback to developers (ideally in just a few minutes) to maintain momentum without getting bogged down by dependency installations.

2. Global / Nightly CI (Comprehensive Testing)

• Trigger: Runs on a scheduled basis (e.g., once a day overnight).
• Scope: Performs the full testing suite. It will build the complete environment, install all third-party software, and run the real integration tests to ensure everything interacts correctly.
• Goal: Catch deeper integration issues, edge cases, and regressions that the mocked PR tests might miss, without blocking active daily development.

Key Benefits

• Faster Iterations: Developers can merge smaller, frequent PRs without waiting on bloated pipelines.
• Resource Efficiency: We save significant compute time by avoiding redundant, heavy software installations on every minor commit.

Discussion
I would love to hear everyone’s thoughts on this pivot. A few specific questions to kick off the discussion:

• What is our preferred approach for mocking the specific software dependencies in the PR pipeline?
• Is a once-a-day schedule sufficient for the Global CI, or should it also trigger automatically on merges to main?
  @alongd @calvinp0

Start by locating the current CI pipeline and inventorying its external software, unit tests, and integration tests. Define separate pull-request and scheduled workflows, decide how external interfaces will be mocked, and document whether the comprehensive workflow also runs on merges to main; done means fast PR feedback and scheduled full integration coverage.