oxidecomputer / oxidecomputer/omicron
Some IP Pool database operations may not be concurrency-safe
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
While working on #8945, I've found a number of database operations handling IP Pools that don't seem particularly safe for concurrent operations. I'll make a fuller list as I find them, but here's one example:
This method checks whether there are any non-deleted Instances with external IPs from the provided IP Pool. The check itself is logically correct, but it seems really hard to avoid TOCTOU issues using it. For example, it's used here:
That method deletes the link between an IP Pool and Silo, after checking that no IPs are allocated using the above. But it's perfectly possible to get the following sequence of events:
- An operator asks to delete the link between a Silo and an IP Pool with no addresses
- The checks here run, and we find there are in fact no used addresses
- Concurrently, a user provisions an instance with an Ephemeral IP from this pool, which issues and completes the address allocation portion of that work
- The
DELETEstatement itself runs, deleting the link between the pool and silo
Now, the user has an IP address allocated out of an IP Pool that is not linked to their Silo, which doesn't seem right. This might be kind of benign, but part of the work in #8945 is to let operators delegate a pool entirely for use by Oxide. If the sequence of events above is followed by "the operator then delegates the pool for internal usage", now we have our services and customer instances sharing a pool. That seems less benign.
We probably want to make the entire "unlink a silo" operation one transaction, which conditionally deletes the link based on whether there are outstanding IPs.
Here's a more complete list of the methods that at least warrant a closer look:
DataStore::ensure_no_instance_ips_outstandingDataStore::ensure_no_floating_ips_outstandingDataStore::ip_pool_fetch_link: This one is used to check that there is a link between a pool and silo, and then allocate an address from that pool.DataStore::ip_pool_fetch_default: Same here, it seems to be used to check for the default pool, and then use that later, even though that default may have changed in the meantime.DataStore::link_default_gatewayandDataStore::unlink_default_gateway: These ones both fetch the Projects, VPCs, and then Internet Gateways based on a Silo, in a series of nested loops. These use theDatastoreCollection::insert_resource()method to conditionally insert records, which means this might be concurrency-safe. But the nested-loops here make me a bit nervous, especially since they can be re-expressed as joins. These deserve a closer look.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start in nexus/db-queries/src/db/datastore/ip_pool.rs with the listed IP pool and gateway methods, then trace their callers and transaction boundaries. Review each check-then-use sequence for concurrent changes; done means the identified operations have an agreed atomicity strategy that prevents stale links, allocations, or gateway relationships.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- database
- Issue type
- Bug
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 30/100