[CI][Releases] Rework Binary building flow, test LLVM/MLIR at least somewhere
Nobody has claimed this yet.
- Dominant language
- C++
- Stars
- 2.2k
- Forks
- 524
- Avg merge
- 3d 2h
- Merged PRs (30d)
- 46
Description
The latest release (https://github.com/llvm/circt/releases/tag/sifive/1/25/0) ran into problems due to upstream LLVM being broken with the compiler we use for building and uploading binaries (https://llvm.org/PR59622).
This has highlighted some issues in the flow that I'd like to discuss regarding how to improve things so we can have a smoother release process and so we can be confident in what we're tagging-- not just that the published binaries are good but also our included non-release-no-promises LLVM version works at least somewhat reasonably on platforms we care about.
A few issues:
-
We publish using a flow/toolchain/environments that are not used elsewhere in our testing, so any problems will only be found during the release process (currently, after a release is tagged). Where reasonable, aligning the tools used with what we test regularly with would help here, as would periodically / strategically checking using the publish tools to get ahead of potential problems. Having the release flow include checking this first is what I suggest to help here to start with, once that's possible to trigger:
-
The "build binaries" workflow cannot be invoked manually, making anyone looking to responsibly tag a release unable to reasonably check if the process "should" work before tagging the release. Since "computers", it'd be better that we re-used the "confirmed to work" results of the flow rather than using that to indicate it most likely won't fail the second time, but given our setup and already constantly-evicting-things-we-need cache strategy it might be simpler to just run it twice. Turnaround is not especially important here, compared to things like the PR CI.
-
In the "build binaries flow" we don't run tests for the LLVM/MLIR that we build and bundle (via linking) for folks to use, which seems .. sub-optimal. Running own integration tests/etc. would be good as well but may require more dependencies or a different build strategy. More on this at the end.
-
Tangentially, we actually don't run LLVM/MLIR tests in any CI AFAICT-- which given our pinned toolchain and LLVM may not need to happen regularly (/me cries in nix), but is something we should consider adding as part of / after bumping LLVM. Hopefully everyone does this themselves already, but that's only for their platform and lacks transparency.
A potential downside here is this requires tests to generally pass, which they should but if are regularly spuriously failing may be a pain for us.
Some of this may need to wait until we have suitable testing power/infrastructure, such as after we're pulled into LLVM proper (where some of this will be unnecessary as well).
Hinted at from (3), WDYT about moving to a unified build for this flow (integration tests, maybe python if we have a packaging story)?
I suggest we also move to leveraging install targets/machinery to drive what's packaged. This would be good to test and maintain as a healthy project anyway (I do this indirectly regularly as part of packaging via nix here but that's unofficial and just the one platform-- that's why I sometimes submit PR's fixing installation issues).
Assuming it works for our target platforms, using the install targets should also make it easy to only install CIRCT (not needing to whitelist the CIRCT bits was the reason we didn't do a unified build when this flow was first introduced).
I'll submit a PR to do the easy solution for (2) shortly.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start by reviewing the build binaries workflow and the release process described in the issue. Define how manual invocation, LLVM/MLIR testing, and install-target-based packaging should fit together, then identify the required CI entry points and release checks. Done means the proposed flow can be validated before tagging and tests the bundled toolchain.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- github-actions
- Domain
- build-system, ci-cd, release, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100