CakeML / CakeML/cakeml

Improving regression parallelism

Open
#785 0 comments 0 reactions 0 assignees View on GitHub
dev experience enhancement low effort medium reward tooling
Dominant language
Standard ML
Stars
1.2k
Forks
104
Avg merge
2d 21h
Merged PRs (30d)
16

Description

Except for the bootstraps, most of the directories have phases where only a few low-memory jobs are running and they could usefully be folded together. However, we still want to be able to limit oversubscription of bootstraps and to have output in a consistent order for easy comparison, which suggests changes to the Holmake logic. (Do we want to try to upstream those changes, fork it, or upstream a refactoring to make the actual build logic into a library and just fork the scheduler/display code?)

I'm thinking something along the lines of:
* each job annotated with a memory limit, which is enforced automatically with --maxheap
* scheduler avoids running jobs simultaneously based on memory limit and a global memory limit
* the regression.cgi page displays a fixed preorder traversal of all Holmake build rules, possibly grouped by directory, and for each rule shows the current status (waiting for dependencies, waiting for CPU/MEM slots, running for X time and memory, succeeded/failed after X time and memory)
* nice to have: create cgroups for more accurate memory accounting

Contributor guide

No contributing guide indexed for this repository

Research direction

Start by examining the Holmake logic and the regression.cgi page mentioned in the issue. First resolve whether the changes should be upstreamed, forked, or split around a library and scheduler/display code. Done would include memory-aware job scheduling and consistent regression status output, with cgroups only if the nice-to-have is included.

Written by the indexing model from the issue text.

Assessment

Domain
build-system, testing
Issue type
Feature
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.