liquidmetal-dev / liquidmetal-dev/flintlock

Actual MVM created state *still* not accurately reported

Open
#326 11 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

kind/bug
Dominant language
Go
Stars
1.5k
Forks
73
Avg merge
17h 24m
Merged PRs (30d)
30

Description

What happened:
Creating an MVM via CAPMVM, the controller logs:

"msg"="Reconciling MicrovmMachine"
"msg"="microvm created"

Checking the mvm:

hammertime -a 147.75.80.57 get -n mvm-test-control-plane-wtfr9 -ns default -s
CREATED

But according to /var/lib/flintlock/vm/default/mvm-test-control-plane-wtfr9/firecracker.stdout:

[    0.166350] Please append a correct "root=" boot option; here are the available partitions:
[    0.166933] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[    0.167554] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-156-generic #163-Ubuntu
[    0.168097] Call Trace:
[    0.168282]  dump_stack+0x6d/0x8b
[    0.168523]  panic+0xe4/0x247
[    0.168743]  mount_block_root+0x1f6/0x2da
[    0.169031]  ? set_debug_rodata+0x17/0x17
[    0.169320]  mount_root+0x38/0x3a
[    0.169560]  prepare_namespace+0x139/0x18e
[    0.169853]  kernel_init_freeable+0x22c/0x259
[    0.170164]  ? rest_init+0xb0/0xb0
[    0.170409]  kernel_init+0xe/0x110
[    0.170660]  ret_from_fork+0x1f/0x40
[    0.170976] Kernel Offset: disabled
[    0.171265] Rebooting in 1 seconds..

So firecracker has failed and we have no idea. Not cool.

(The given error is just me misconfiguring, so don't worry about that.)

What did you expect to happen:

Capmvm should have been told there was something wrong, it would not be ideal for users to ssh on to read logs, so flintlock needs to know whether a create has actually created.

How to reproduce it:
Don't need capmvm, can probably make it fail by messing up the boot command.

Anything else you would like to add:
I know we hoped the config file would help with this, but either we were wrong or maybe we are missing something.

A recent pr made it so that if anything before firecracker fails, the state will stay as pending, but beyond seeing that the call to FC api succeeds, we do not know the result.

By the time i look at the process, i can see that no firecracker is running. So it started and died and we dont check. Maybe we need to kick up the supervisor, feels important for product to get this info back to the user, and even if we check the pid is running after create it will die after that.

Environment:

  • flintlock version: built from main at 7213b9c
  • containerd version:
  • OS (e.g. from /etc/os-release):

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the controller's create path and the supervisor mentioned in the issue, then reproduce a failed Firecracker boot by using an invalid boot command. Check how the Firecracker API result and later process failure affect the reported MVM state. Done means a VM that starts and then fails is reported as failed rather than CREATED, without requiring users to inspect logs over SSH.

Written by the indexing model from the issue text.

Assessment

Tech stack
go
Domain
backend, infrastructure
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.