oxidecomputer / oxidecomputer/hubris
RoT failed to update with UpdateNotStarted
Open
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 3.6k
- Forks
- 239
- Avg merge
- 1d 12h
- Merged PRs (30d)
- 23
Description
In updating the rack, we had two sleds (cubby 9 and cubby 26) fail with UpdateNotStarted:
The dump from the SP on sled 26 isn't particularly revealing:
$ humility -d ./hubris.core.sled-26.failed-to-update-rot ringbuf control_plane_agent
humility: attached to dump
humility: ring buffer task_control_plane_agent::__RINGBUF in control_plane_agent:
NDX LINE GEN COUNT PAYLOAD
4 497 55377 13 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x0, 0x2 ])), port: 0x56ce, size: 0x20, vid: 0x301 })
5 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xa, vid: 0x302 })
6 84 55377 1 MgsMessage(SpState)
7 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
8 836 55377 1 MgsMessage(Inventory)
9 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
10 836 55377 1 MgsMessage(Inventory)
11 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
12 836 55377 1 MgsMessage(Inventory)
13 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
14 836 55377 1 MgsMessage(Inventory)
15 497 55377 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
0 836 55378 1 MgsMessage(Inventory)
1 497 55378 1 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0xe, vid: 0x302 })
2 836 55378 1 MgsMessage(Inventory)
3 497 55378 13 Rx(UdpMetadata { addr: Ipv6(Ipv6Address([ 0xfe, 0x80, 0x0, 0x0, 0x0, 0x0, 0x0, 0x0, 0xaa, 0x40, 0x25, 0xff, 0xfe, 0x5, 0x4, 0x2 ])), port: 0x56ce, size: 0x20, vid: 0x302 })
humility: ring buffer task_control_plane_agent::update::rot::__RINGBUF in control_plane_agent:
NDX LINE GEN COUNT PAYLOAD
38 190 8 1 WriteOneBlock(0xec, 0x200, 0x200)
39 190 8 1 WriteOneBlock(0xed, 0x200, 0x200)
40 138 8 1 IngestChunkInput { offset: 0x1dd8a, len: 0x3d2 }
41 161 8 1 IngestChunkState { offset: 0x1dc00, len: 0x18a }
42 190 8 1 WriteOneBlock(0xee, 0x200, 0x200)
43 190 8 1 WriteOneBlock(0xef, 0x200, 0x200)
44 138 8 1 IngestChunkInput { offset: 0x1e15c, len: 0x3d2 }
45 161 8 1 IngestChunkState { offset: 0x1e000, len: 0x15c }
46 190 8 1 WriteOneBlock(0xf0, 0x200, 0x200)
47 190 8 1 WriteOneBlock(0xf1, 0x200, 0x200)
48 138 8 1 IngestChunkInput { offset: 0x1e52e, len: 0x3d2 }
49 161 8 1 IngestChunkState { offset: 0x1e400, len: 0x12e }
50 190 8 1 WriteOneBlock(0xf2, 0x200, 0x200)
51 190 8 1 WriteOneBlock(0xf3, 0x200, 0x200)
52 138 8 1 IngestChunkInput { offset: 0x1e900, len: 0x3d2 }
53 161 8 1 IngestChunkState { offset: 0x1e800, len: 0x100 }
54 190 8 1 WriteOneBlock(0xf4, 0x200, 0x200)
55 190 8 1 WriteOneBlock(0xf5, 0x200, 0x200)
56 138 8 1 IngestChunkInput { offset: 0x1ecd2, len: 0x3d2 }
57 161 8 1 IngestChunkState { offset: 0x1ec00, len: 0xd2 }
58 190 8 1 WriteOneBlock(0xf6, 0x200, 0x200)
59 190 8 1 WriteOneBlock(0xf7, 0x200, 0x200)
60 138 8 1 IngestChunkInput { offset: 0x1f0a4, len: 0x3d2 }
61 161 8 1 IngestChunkState { offset: 0x1f000, len: 0xa4 }
62 190 8 1 WriteOneBlock(0xf8, 0x200, 0x200)
63 190 8 1 WriteOneBlock(0xf9, 0x200, 0x200)
0 138 9 1 IngestChunkInput { offset: 0x1f476, len: 0x3d2 }
1 161 9 1 IngestChunkState { offset: 0x1f400, len: 0x76 }
2 190 9 1 WriteOneBlock(0xfa, 0x200, 0x200)
3 190 9 1 WriteOneBlock(0xfb, 0x200, 0x200)
4 138 9 1 IngestChunkInput { offset: 0x1f848, len: 0x3d2 }
5 161 9 1 IngestChunkState { offset: 0x1f800, len: 0x48 }
6 190 9 1 WriteOneBlock(0xfc, 0x200, 0x200)
7 190 9 1 WriteOneBlock(0xfd, 0x200, 0x200)
8 138 9 1 IngestChunkInput { offset: 0x1fc1a, len: 0x3d2 }
9 161 9 1 IngestChunkState { offset: 0x1fc00, len: 0x1a }
10 190 9 1 WriteOneBlock(0xfe, 0x200, 0x200)
11 138 9 1 IngestChunkInput { offset: 0x1ffec, len: 0x3d2 }
12 161 9 1 IngestChunkState { offset: 0x1fe00, len: 0x1ec }
13 190 9 1 WriteOneBlock(0xff, 0x200, 0x200)
14 190 9 1 WriteOneBlock(0x100, 0x200, 0x200)
15 138 9 1 IngestChunkInput { offset: 0x203be, len: 0x3d2 }
16 161 9 1 IngestChunkState { offset: 0x20200, len: 0x1be }
17 190 9 1 WriteOneBlock(0x101, 0x200, 0x200)
18 190 9 1 WriteOneBlock(0x102, 0x200, 0x200)
19 138 9 1 IngestChunkInput { offset: 0x20790, len: 0x3d2 }
20 161 9 1 IngestChunkState { offset: 0x20600, len: 0x190 }
21 190 9 1 WriteOneBlock(0x103, 0x200, 0x200)
22 190 9 1 WriteOneBlock(0x104, 0x200, 0x200)
23 138 9 1 IngestChunkInput { offset: 0x20b62, len: 0x3d2 }
24 161 9 1 IngestChunkState { offset: 0x20a00, len: 0x162 }
25 190 9 1 WriteOneBlock(0x105, 0x200, 0x200)
26 190 9 1 WriteOneBlock(0x106, 0x200, 0x200)
27 138 9 1 IngestChunkInput { offset: 0x20f34, len: 0x3d2 }
28 161 9 1 IngestChunkState { offset: 0x20e00, len: 0x134 }
29 190 9 1 WriteOneBlock(0x107, 0x200, 0x200)
30 190 9 1 WriteOneBlock(0x108, 0x200, 0x200)
31 138 9 1 IngestChunkInput { offset: 0x21306, len: 0x3d2 }
32 161 9 1 IngestChunkState { offset: 0x21200, len: 0x106 }
33 190 9 1 WriteOneBlock(0x109, 0x200, 0x200)
34 190 9 1 WriteOneBlock(0x10a, 0x200, 0x200)
35 138 9 1 IngestChunkInput { offset: 0x216d8, len: 0x3d2 }
36 161 9 1 IngestChunkState { offset: 0x21600, len: 0xd8 }
37 190 9 1 WriteOneBlock(0x10b, 0x200, 0x200)
Reattempting the upgrade succeeded.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the control_plane_agent::update::rot ring buffer shown in the issue and compare the failed dump with the successful retry. Trace how UpdateNotStarted is reported around the listed write and ingest events. Done would require identifying the cause and a corrective change, but the issue names no source file or test.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- embedded-iot, operating-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 28/100