ocurrent / ocurrent/ocluster

`zfs` and `chown` subprocesses calls are failing to `internal error` in ocluster-worker

Open
#248 3 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
OCaml
Stars
42
Forks
20
PR merge metrics
No merged PRs in 30d

Description

Noticed in https://github.com/tarides/infrastructure/issues/375#issuecomment-2329726772

ocluster-worker: internal error, uncaught exception:
                 Failure("\"zfs\" \"create\" \"--\" \"obuilder/state\" failed with exit status 1")
                 
2024-09-04 20:58.56         worker [INFO] Prune threshold not set and docker max df size is not. Will not check for low disk-space!
cannot open 'obuilder/cache-tmp': dataset does not exist
cannot open 'obuilder/state': dataset does not exist
2024-09-04 20:58.56    application [INFO] Exec "zfs" "create" "--" "obuilder/state"
cannot create 'obuilder/state': no such pool 'obuilder'
ocluster-worker: internal error, uncaught exception:
                 Failure("\"zfs\" \"create\" \"--\" \"obuilder/state\" failed with exit status 1")
                 
2024-09-04 20:59.06         worker [INFO] Prune threshold not set and docker max df size is not. Will not check for low disk-space!
2024-09-04 20:59.06    application [INFO] Exec "zfs" "destroy" "-R" "-f" "--" "obuilder/cache-tmp"
2024-09-04 20:59.07    application [INFO] Exec "chown" "0:0" "/Volumes/obuilder/state"
chown: /Volumes/obuilder/state: No such file or directory
ocluster-worker: internal error, uncaught exception:
                 Failure("\"chown\" \"0:0\" \"/Volumes/obuilder/state\" failed with exit status 1")
ocluster-worker: internal error, uncaught exception:
                 Failure("\"chown\" \"0:0\" \"/Volumes/obuilder/state\" failed with exit status 1")

administrator@m1-worker-03 ~ % grep -A 2 'internal error' ./ocluster.log | tail -n 10
                 Failure("\"zfs\" \"create\" \"--\" \"obuilder/state\" failed with exit status 1")

I suspect this may be causing, contributing to, or masking problems that lead the macos builders to stop being able to build jobs. But proper error handling for these sub process calls should be put in place regardless.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by tracing the ocluster-worker subprocess handling for the failing zfs and chown calls, using the reported logs as the reproduction context. Determine how these non-zero exits are currently propagated and define the expected handling for missing datasets or paths; done means the failures no longer surface as uncaught internal errors and the relevant behavior is covered by tests, if existing tests are found.

Written by the indexing model from the issue text.

Assessment

Tech stack
macos, ocaml
Domain
devops, operating-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.