llvm / llvm/llvm-project

Stop using -march=native in CI

Open
#196,491 1 comment 0 reactions 0 assignees View on GitHub
libc
Dominant language
LLVM
Stars
40.5k
Forks
18.7k
PR merge metrics
PR metrics pending

Description

-march=native is causing two problems:

1) Caching of artifacts generated this way cause SIGILL when running on older processor

## Evidence

Three consecutive runs of the same fwide PR (https://github.com/llvm/llvm-project/pull/196157), same code:

| Run | Azure Region | Cache Hits | Cache Misses | Result |
|-----|-------------|-----------|-------------|--------|
|
[25512875679](https://github.com/llvm/llvm-project/actions/runs/25512875679/job/74876008545)
| westus3 | 9 | 5354 | PASS |
|
[25524024922](https://github.com/llvm/llvm-project/actions/runs/25524024922/job/74916241365)
| northcentralus | 5345 | 0 | CRASH |
|
[25524839613](https://github.com/llvm/llvm-project/actions/runs/25524839613/job/74965830435)
| westus | 5345 | 0 | CRASH |

The first run had a nearly empty cache and compiled everything locally
(0.17% hit rate). An intermediate [syscall-unistd
run](https://github.com/llvm/llvm-project/actions/runs/25517783708/job/74893495220)
in eastus then populated the cache with object files compiled for that
region's CPU. Subsequent runs on different hardware got 100% cache hits
and crashed because the cached `.o` files use instructions their CPUs
lack.

This was mitigated by https://github.com/llvm/llvm-project/commit/a33ba5f0948c2e6c365468a113a7d1ae5afb37b5

2) Inconsistency of builds

CI aims to be 100% reproducible, and having the output from the compiler change based on what machine is silently picked isn't best practice.

That said, we almost certainly want to test on the range of CPUs since clearly different machine code is being generated for each target. We should have adequate test coverage across the range of CPUs available to us.

The mitigation used almost certainly should be rolled back after we stop using -march=native

Contributor guide

Open the contributing guide

Research direction

Start by locating the CI configuration that adds -march=native and reviewing commit a33ba5f0948c2e6c365468a113a7d1ae5afb37b5, along with the referenced fwide and syscall-unistd runs. Check how cached object files are keyed across Azure regions and how CPU coverage is currently exercised. Done means CI builds are reproducible and cached artifacts do not require unsupported host instructions, while CPU variation remains tested.

Written by the indexing model from the issue text.

Assessment

Tech stack
azure, github-actions
Domain
build-system, ci-cd, infrastructure
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.