ocurrent / ocurrent/ocluster

Trying to understand the bottleneck that some workers experience

Open
#114 8 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
OCaml
Stars
42
Forks
20
PR merge metrics
No merged PRs in 30d

Description

I've been running tests on pisto exclusively the past few days and I'm a bit surprised by how many jobs actually are running at once.
In my case, I have:

  • an opam-health-check instance giving exactly 80 jobs to the test pool
  • the 80 jobs fill the worker capacity to its maximum

However most of the jobs stop at:

[...]
---> using "486382d4d06e1d3cd81a0566311ede22726942ab1a24bce8a41318c6a5717b58" from cache

/: (run (cache (opam-archives (target /home/opam/.opam/download-cache)))
        (network host)
        (shell  "<cmd>"))

I've only seen a maximum of 23 out of the 79 jobs that actually start the said command, I'm not sure what's happening to the rest (runc isn't even started).
Maybe there is some kind of IO bottleneck partially caused by https://github.com/ocaml/opam/issues/4586 + maybe the opam-archives cache might be too big and btrfs is struggling to pull it (?)

The load average of the machine in this state is around 15% so if there is a bottleneck it must be some kind of IO or syscall bottleneck

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Reproduce the reported workload with an opam-health-check instance supplying 80 jobs, then inspect why only about 23 workers reach the command while others stop at the cache setup. Compare the opam-archives cache, btrfs, runc startup, and IO or syscall activity; done means identifying the bottleneck with supporting evidence.

Written by the indexing model from the issue text.

Assessment

Tech stack
ocaml
Domain
ci-cd, infrastructure, performance
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
20/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.