Trying to understand the bottleneck that some workers experience
Nobody has claimed this yet.
- Dominant language
- OCaml
- Stars
- 42
- Forks
- 20
- PR merge metrics
- No merged PRs in 30d
Description
I've been running tests on pisto exclusively the past few days and I'm a bit surprised by how many jobs actually are running at once.
In my case, I have:
- an opam-health-check instance giving exactly 80 jobs to the test pool
- the 80 jobs fill the worker capacity to its maximum
However most of the jobs stop at:
[...]
---> using "486382d4d06e1d3cd81a0566311ede22726942ab1a24bce8a41318c6a5717b58" from cache
/: (run (cache (opam-archives (target /home/opam/.opam/download-cache)))
(network host)
(shell "<cmd>"))
I've only seen a maximum of 23 out of the 79 jobs that actually start the said command, I'm not sure what's happening to the rest (runc isn't even started).
Maybe there is some kind of IO bottleneck partially caused by https://github.com/ocaml/opam/issues/4586 + maybe the opam-archives cache might be too big and btrfs is struggling to pull it (?)
The load average of the machine in this state is around 15% so if there is a bottleneck it must be some kind of IO or syscall bottleneck
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Reproduce the reported workload with an opam-health-check instance supplying 80 jobs, then inspect why only about 23 workers reach the command while others stop at the cache setup. Compare the opam-archives cache, btrfs, runc startup, and IO or syscall activity; done means identifying the bottleneck with supporting evidence.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- ocaml
- Domain
- ci-cd, infrastructure, performance
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 20/100