oxidecomputer / oxidecomputer/omicron
vpc_create saga failed after recovery: "route4" node not idempotent
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
Like #6069, I saw this while testing #6063. Out of 31 project-create sagas (which include a vpc-create subsaga) that were in-progress during a Nexus restart and then recovered, one of them failed like this:
18:32:33.350Z WARN 333518da-5601-43e8-b9bd-f675571c5797 (ServerContext): saga finished
action_error_node_name = "route4"
action_error_source = ActionFailed { source_error: Object {"ObjectAlreadyExists": Object {"object_name": String("default-v4"), "type_name": String("router-route")}} }
file = /home/dap/.cargo/registry/src/index.crates.io-6f17d22bba15001f/steno-0.4.1/src/sec.rs:1044
result = failure
saga_id = c4162dd2-c1b6-4ce8-b736-52d1133ff946
saga_name = project-create
sec_id = 333518da-5601-43e8-b9bd-f675571c5797
undo_result = success
This is the "route4" action:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/src/app/sagas/vpc_create.rs#L48-L51
implemented with svc_create_v4_route:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/src/app/sagas/vpc_create.rs#L231-L239
which is trying to insert an item into the router_route table. Based on the ErrorHandler::Conflict, this code is expecting a possible conflict on the route's name. That's deafult-v4, all the way up the stack. And there's a unique index on router_route for (vpc_router_id, name). So I think this would all be expected if we started running the action before the crash, we inserted the row, then Nexus crashed before the action formally completed, then Nexus recovered the saga, and then Steno re-ran the action.
See also #6070.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start with the route4 action in nexus/src/app/sagas/vpc_create.rs, then follow its calls into nexus/db-queries/src/db/datastore/vpc.rs and the router_route insert. Review the recovery scenario described in the issue and determine how the action should behave when the route already exists after a Nexus restart. Done means a recovered project-create saga does not fail with ObjectAlreadyExists for the existing default-v4 route.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, databases, distributed-systems
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 35/100