coreml_static_int8 operators suite never finishes on macOS (hits the 180 min cap)
Nobody has claimed this yet.
- Dominant language
- Python
- Stars
- 5k
- Forks
- 1.2k
- Avg merge
- 2d 10h
- Merged PRs (30d)
- 581
Description
The coreml_static_int8 half of test-backend-macos (operators suite) has never completed. It runs to the 180 minute job cap and gets cancelled, so it produces no signal at all.
Measured on a recent main commit:
| flow / suite | duration |
|---|---|
| coreml / models | 12 min |
| coreml / operators | 11 min |
| coreml_static_int8 / models | 25 min |
| coreml_static_int8 / operators | 181 min, cancelled at the cap |
Both flows are built by the same _create_coreml_flow in backends/test/suite/flows/coreml.py. coreml_static_int8 is the identical models + operators suite with quantize=True and minimum_deployment_target=iOS17. So the cost is a full PT2E prepare / calibrate / convert pass on every single test case, not larger weights.
#21609 moved this flow to the nightly schedule, so it no longer consumes the shared macOS pool on every commit. That limits the damage but does not fix it: the nightly run will keep hitting the same cap, and the int8 flow will keep reporting nothing.
Possible directions, roughly in order of effort:
- Shard the operators suite across several jobs so each one finishes inside the cap.
- Reuse a single quantized artifact across cases where the operator under test allows it, instead of quantizing per case.
- Trim the int8 case list to the operators where quantization actually changes the lowering.
Until one of those lands, the int8 operators coverage should be treated as absent rather than passing.
cc @kimishpatel @YifanShenSZ @cymbalrush @metascroy
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in backends/test/suite/flows/coreml.py, especially _create_coreml_flow, and compare the coreml_static_int8 operators flow with the other listed suites. Investigate sharding, artifact reuse, or case trimming, then verify that the int8 operators coverage produces signal and finishes within the 180-minute macOS job cap.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- python
- Domain
- ci-cd, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Quiet
- Clarity
- Mostly clear
- Newbie friendliness
- 48/100