oxidecomputer / oxidecomputer/maghemite

test flake: occasional hang in falcon based tests

Open
#679 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Bug CI
Dominant language
Rust
Stars
94
Forks
6
Avg merge
1d 8h
Merged PRs (30d)
12

Description

It would seem that sometimes tests using falcon will occasionally hang indefinitely -- waiting on a manual job cancellation or a buildomat timeout to fail.
e.g.
https://buildomat.eng.oxide.computer/wg/0/details/01KMNFDTJVCTFY7728BSTC4XFB/zAyqoVXcFuT46hzSpuBpo3GaQ7rpHk7Dt2TNG5YZsDTe9Ocy/01KMNFF3HR8BRKC2ER3S4RAPFX

In this example, pfexec ./falcon-lab run trio-unnumbered was executed to kick off the falcon-lab test. The last log written by falcon was this:

2026-03-26T16:51:48.361Z	Mar 26 16:51:48.348 INFO cr2: logged out

which sat for about an hour and a half until I manually cancelled the job:

2026-03-26T18:24:32.741Z	job cancelled

This issue is to track similar flakes where falcon hangs for unreasonable amounts of time.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start by reproducing the falcon-lab test with pfexec ./falcon-lab run trio-unnumbered and review the linked Buildomat run. Trace what happens after the final cr2: logged out message; done means the hang's cause is identified and the affected test no longer waits indefinitely for cancellation or timeout.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
networking, testing-qa
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Quiet
Clarity
Needs clarification
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.