liquidmetal-dev / liquidmetal-dev/flintlock
Actual MVM created state *still* not accurately reported
Nobody has claimed this yet.
- Dominant language
- Go
- Stars
- 1.5k
- Forks
- 73
- Avg merge
- 17h 24m
- Merged PRs (30d)
- 30
Description
What happened:
Creating an MVM via CAPMVM, the controller logs:
"msg"="Reconciling MicrovmMachine"
"msg"="microvm created"
Checking the mvm:
hammertime -a 147.75.80.57 get -n mvm-test-control-plane-wtfr9 -ns default -s
CREATED
But according to /var/lib/flintlock/vm/default/mvm-test-control-plane-wtfr9/firecracker.stdout:
[ 0.166350] Please append a correct "root=" boot option; here are the available partitions:
[ 0.166933] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(0,0)
[ 0.167554] CPU: 0 PID: 1 Comm: swapper/0 Not tainted 4.15.0-156-generic #163-Ubuntu
[ 0.168097] Call Trace:
[ 0.168282] dump_stack+0x6d/0x8b
[ 0.168523] panic+0xe4/0x247
[ 0.168743] mount_block_root+0x1f6/0x2da
[ 0.169031] ? set_debug_rodata+0x17/0x17
[ 0.169320] mount_root+0x38/0x3a
[ 0.169560] prepare_namespace+0x139/0x18e
[ 0.169853] kernel_init_freeable+0x22c/0x259
[ 0.170164] ? rest_init+0xb0/0xb0
[ 0.170409] kernel_init+0xe/0x110
[ 0.170660] ret_from_fork+0x1f/0x40
[ 0.170976] Kernel Offset: disabled
[ 0.171265] Rebooting in 1 seconds..
So firecracker has failed and we have no idea. Not cool.
(The given error is just me misconfiguring, so don't worry about that.)
What did you expect to happen:
Capmvm should have been told there was something wrong, it would not be ideal for users to ssh on to read logs, so flintlock needs to know whether a create has actually created.
How to reproduce it:
Don't need capmvm, can probably make it fail by messing up the boot command.
Anything else you would like to add:
I know we hoped the config file would help with this, but either we were wrong or maybe we are missing something.
A recent pr made it so that if anything before firecracker fails, the state will stay as pending, but beyond seeing that the call to FC api succeeds, we do not know the result.
By the time i look at the process, i can see that no firecracker is running. So it started and died and we dont check. Maybe we need to kick up the supervisor, feels important for product to get this info back to the user, and even if we check the pid is running after create it will die after that.
Environment:
- flintlock version: built from main at 7213b9c
- containerd version:
- OS (e.g. from
/etc/os-release):
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the controller's create path and the supervisor mentioned in the issue, then reproduce a failed Firecracker boot by using an invalid boot command. Check how the Firecracker API result and later process failure affect the reported MVM state. Done means a VM that starts and then fails is reported as failed rather than CREATED, without requiring users to inspect logs over SSH.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- go
- Domain
- backend, infrastructure
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 25/100