oxidecomputer / oxidecomputer/buildomat
worker agent on linux might be getting oomkilled when it's the job's fault instead
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 72
- Forks
- 5
- PR merge metrics
- No merged PRs in 30d
Description
I think I'm seeing a worker agent experienced a fatal error; aborting job error on this run because the worker agent is getting oomkilled on Linux:
This is occurring after TestLint/TestErrCheck, which runs a command that completely exhausts the 32 GB of RAM on a machine I'm debugging this on. I wonder if the entire cgroup is getting axed and not just the underlying job process.
Does the agent set its oomkiller priority at all? (I think there's like three ways to do this now on Linux because of course there is.)
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the linked build run and trace how the Linux worker agent handles cgroups and out-of-memory events around TestLint/TestErrCheck. Determine whether the agent or the job process is killed, then document or fix the behavior so a job's memory exhaustion does not incorrectly appear as an agent fatal error.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- linux, rust
- Domain
- build-system, infrastructure
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 35/100