extremely slow schema updates after MUPdate

Open
#10,475 4 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Assessment

Difficulty
4/5
Estimated time
3-5 days
Newbie friendliness
35/100
Issue type
Bug
Clarity
Needs clarification
Activity status
Active
Tech stack
rust
Domain
backend, databases

Research direction

Start with the Nexus logs and the DTrace query output to trace the slow schema migration, then compare them with the CockroachDB jobs table and logs. Determine why the migration makes forward progress so slowly after an R18-to-R19 MUPdate and verify that Nexus completes the schema update without appearing unavailable before opening its listening sockets.

Written by the indexing model from the issue text.

Description

On a customer system, after a MUPdate, we found that the normal schema migration that Nexus does was proceeding very slowly. In fact, we initially thought Nexus hadn't come up because:

$ omdb nexus update-status
note: Nexus URL not specified.  Will pick one from DNS.
note: using DNS from system config (typically /etc/resolv.conf)
note: (if this is not right, use --dns-server to specify an alternate DNS server)
warning: failed to connect to NexusLockstep at ...:12232: connect "...:12232": Connection refused (os error 146)
warning: failed to connect to NexusLockstep at ...:12232: connect "...:12232": Connection refused (os error 146)
warning: failed to connect to NexusLockstep at ...:12232: connect "...:12232": Connection refused (os error 146)
Error: failed to connect to any instances of NexusLockstep

We looked at it for a while before realizing that Nexus was making forward progress, just very slowly. Nexus completes schema migrations before opening up its listening sockets, which is why we got the error above.

For reference:

  • I believe this was a MUPdate from R18 to R19.
  • The rack was parked for MUPdate around 2026-04-30T16:00:00Z, or 9am PT.

We were able to tell that schema upgrade steps were happening very slowly but it wasn't clear why. We looked at various data:

  • Nexus logs
  • output from a DTrace script showing queries executing by Nexus nodes
  • the jobs table reported by CockroachDB
  • CockroachDB logs

More details coming.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

More from oxidecomputer/omicron

All issues in oxidecomputer/omicron

Similar issues

More Rust issues

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.