Improving regression parallelism
- Dominant language
- Standard ML
- Stars
- 1.2k
- Forks
- 104
- Avg merge
- 2d 21h
- Merged PRs (30d)
- 16
Description
Except for the bootstraps, most of the directories have phases where only a few low-memory jobs are running and they could usefully be folded together. However, we still want to be able to limit oversubscription of bootstraps and to have output in a consistent order for easy comparison, which suggests changes to the Holmake logic. (Do we want to try to upstream those changes, fork it, or upstream a refactoring to make the actual build logic into a library and just fork the scheduler/display code?)
I'm thinking something along the lines of:
* each job annotated with a memory limit, which is enforced automatically with --maxheap
* scheduler avoids running jobs simultaneously based on memory limit and a global memory limit
* the regression.cgi page displays a fixed preorder traversal of all Holmake build rules, possibly grouped by directory, and for each rule shows the current status (waiting for dependencies, waiting for CPU/MEM slots, running for X time and memory, succeeded/failed after X time and memory)
* nice to have: create cgroups for more accurate memory accounting
Contributor guide
No contributing guide indexed for this repository
Research direction
Start by examining the Holmake logic and the regression.cgi page mentioned in the issue. First resolve whether the changes should be upstreamed, forked, or split around a library and scheduler/display code. Done would include memory-aware job scheduling and consistent regression status output, with cgroups only if the nice-to-have is included.
Written by the indexing model from the issue text.
Assessment
- Domain
- build-system, testing
- Issue type
- Feature
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100