oxidecomputer / oxidecomputer/buildomat

lab factory missed exit of `ipmitool` child process

Open
#18 0 comments 0 reactions 1 assignee View on GitHub

@jclulow is already working on this.

Since Mar 29, 2023.

Dominant language
Rust
Stars
72
Forks
5
PR merge metrics
No merged PRs in 30d

Description

We see this in the log:

Mar 27 01:55:54.348 INFO [sock/1945] serial: "SOL session closed by BMC", name: factory-lab

And nothing about serial for sock after that. There is no message specifically about the process going away that I can see, but it is indeed gone:

jclulow@catacomb ~
$ ptree buildomat
7675   /opt/buildomat/lib/buildomat-factory-lab -f config-lab.toml -d lab.sqlite3
  7678   <defunct>
  7679   /usr/sbin/ipmitool -C 3 -I lanplus -H buskin.lom.eng.oxide.computer -U ...

A few things:

  • we should log at least the pid of the ipmitool child process; if we are today, I could not see it in there.
  • find out why we missed the process exiting; we clearly have not called wait on the child as it has not been reaped
  • belt and suspenders, we should treat an emission from ipmitool of "SOL session closed by BMC" on a line by itself as the start of some timeout after which we should forcefully chuck out the process if we have heard nothing else from it

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Assessment

This issue has not been assessed yet.

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.