oxidecomputer / oxidecomputer/omicron

Improve zone-bundle creation performance to reduce stop-instance response times and avoid timeout situations

Open
#5,236 1 comment 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

Dominant language
Rust
Stars
572
Forks
97
Avg merge
2d 12h
Merged PRs (30d)
96

Description

#5235 uncovered a potential need for zone bundle processing performance improvement. The failure mode may be more common than we think when user spins up a large number of long-running worker instances and spins them down en masse, resulting in concurrent zone-bundle requests on propolis zones that all have a large number of propolis log files to be tar-ed up.

Here are the relevant comments from the customer ticket that provide more context to the possible solutions to this issue:

@gjcolombo

All zone bundle collections are serialized with respect to each other: instances share a single ZoneBundler that's owned by the InstanceManager, and that has an Arc<Mutex<zone_bundle::Inner>>, which lock is required to collect a bundle.

@bnaecker

Bundle generation probably can be faster, since I did nothing to guarantee their performance. It's copying a bunch of log files from the U.2s into a tarball, which is probably what takes the most time, since there tend to be a lot of files given our log rotation policy. I don't think that itself can be parallelized naively, but I'm sure there's something we can do.

The other thing that may take time is running commands in the zone. It shouldn't take too long, since we run pretty standard things like svcs and netstat -an, but I imagine there's always a way for that to take longer than we want under Tokio.

As for parallelization, I'm not sure. The zone itself needs to be owned while we're taking the bundle (at least, without some invasive changes to the code), to ensure it doesn't disappear. When I wrote this, there was a single lock around the whole sled-agent map for instances, which we also need to take for this to work. I think that stuff may have changed recently, with Sean's work to put the instance runner in a separate task, but I've not looked. There may be opportunities for parallelism now.

Contributor guide

Open the contributing guide

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Trace ZoneBundler ownership through InstanceManager and the zone_bundle::Inner mutex, then inspect bundle collection, log-file tar creation, and commands run in the zone. Compare the sled-agent instance map and instance-runner ownership mentioned in the issue, and define completion as faster concurrent zone-bundle requests that avoid stop-instance timeouts.

Written by the indexing model from the issue text.

Assessment

Tech stack
rust
Domain
backend, performance
Issue type
Refactor
Difficulty
5/5
Estimated time
Over a week
Activity status
Stale
Clarity
Needs clarification
Newbie friendliness
25/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.