oxidecomputer / oxidecomputer/omicron

vpc_create saga failed after recovery: "route4" node not idempotent

Open
#6,071 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

Like #6069, I saw this while testing #6063. Out of 31 project-create sagas (which include a vpc-create subsaga) that were in-progress during a Nexus restart and then recovered, one of them failed like this:

18:32:33.350Z WARN 333518da-5601-43e8-b9bd-f675571c5797 (ServerContext): saga finished
    action_error_node_name = "route4"
    action_error_source = ActionFailed { source_error: Object {"ObjectAlreadyExists": Object {"object_name": String("default-v4"), "type_name": String("router-route")}} }
    file = /home/dap/.cargo/registry/src/index.crates.io-6f17d22bba15001f/steno-0.4.1/src/sec.rs:1044
    result = failure
    saga_id = c4162dd2-c1b6-4ce8-b736-52d1133ff946
    saga_name = project-create
    sec_id = 333518da-5601-43e8-b9bd-f675571c5797
    undo_result = success

This is the "route4" action:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/src/app/sagas/vpc_create.rs#L48-L51

implemented with svc_create_v4_route:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/src/app/sagas/vpc_create.rs#L231-L239

which calls:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/src/app/sagas/vpc_create.rs#L265-L300

which calls:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/db-queries/src/db/datastore/vpc.rs#L1214-L1228

which calls:
https://github.com/oxidecomputer/omicron/blob/fe60eb9c20b105c023e68d27cdb6605e4211fe2c/nexus/db-queries/src/db/datastore/vpc.rs#L1230-L1257

which is trying to insert an item into the router_route table. Based on the ErrorHandler::Conflict, this code is expecting a possible conflict on the route's name. That's deafult-v4, all the way up the stack. And there's a unique index on router_route for (vpc_router_id, name). So I think this would all be expected if we started running the action before the crash, we inserted the row, then Nexus crashed before the action formally completed, then Nexus recovered the saga, and then Steno re-ran the action.

See also #6070.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start with the route4 action in nexus/src/app/sagas/vpc_create.rs, then follow its calls into nexus/db-queries/src/db/datastore/vpc.rs and the router_route insert. Review the recovery scenario described in the issue and determine how the action should behave when the route already exists after a Nexus restart. Done means a recovered project-create saga does not fail with ObjectAlreadyExists for the existing default-v4 route.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, databases, distributed-systems
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
35/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.