oxidecomputer / oxidecomputer/omicron

Some IP Pool database operations may not be concurrency-safe

Open
#8,992 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

While working on #8945, I've found a number of database operations handling IP Pools that don't seem particularly safe for concurrent operations. I'll make a fuller list as I find them, but here's one example:

https://github.com/oxidecomputer/omicron/blob/94f0731fcc7686fd6b75dba1a2b0bc3e014e72d6/nexus/db-queries/src/db/datastore/ip_pool.rs#L959

This method checks whether there are any non-deleted Instances with external IPs from the provided IP Pool. The check itself is logically correct, but it seems really hard to avoid TOCTOU issues using it. For example, it's used here:

https://github.com/oxidecomputer/omicron/blob/94f0731fcc7686fd6b75dba1a2b0bc3e014e72d6/nexus/db-queries/src/db/datastore/ip_pool.rs#L1050-L1083

That method deletes the link between an IP Pool and Silo, after checking that no IPs are allocated using the above. But it's perfectly possible to get the following sequence of events:

  • An operator asks to delete the link between a Silo and an IP Pool with no addresses
  • The checks here run, and we find there are in fact no used addresses
  • Concurrently, a user provisions an instance with an Ephemeral IP from this pool, which issues and completes the address allocation portion of that work
  • The DELETE statement itself runs, deleting the link between the pool and silo

Now, the user has an IP address allocated out of an IP Pool that is not linked to their Silo, which doesn't seem right. This might be kind of benign, but part of the work in #8945 is to let operators delegate a pool entirely for use by Oxide. If the sequence of events above is followed by "the operator then delegates the pool for internal usage", now we have our services and customer instances sharing a pool. That seems less benign.

We probably want to make the entire "unlink a silo" operation one transaction, which conditionally deletes the link based on whether there are outstanding IPs.

Here's a more complete list of the methods that at least warrant a closer look:

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start in nexus/db-queries/src/db/datastore/ip_pool.rs with the listed IP pool and gateway methods, then trace their callers and transaction boundaries. Review each check-then-use sequence for concurrent changes; done means the identified operations have an agreed atomicity strategy that prevents stale links, allocations, or gateway relationships.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
database
Issue type
Bug
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
30/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.