oxidecomputer / oxidecomputer/omicron
Improve zone-bundle creation performance to reduce stop-instance response times and avoid timeout situations
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 572
- Forks
- 97
- Avg merge
- 2d 12h
- Merged PRs (30d)
- 96
Description
#5235 uncovered a potential need for zone bundle processing performance improvement. The failure mode may be more common than we think when user spins up a large number of long-running worker instances and spins them down en masse, resulting in concurrent zone-bundle requests on propolis zones that all have a large number of propolis log files to be tar-ed up.
Here are the relevant comments from the customer ticket that provide more context to the possible solutions to this issue:
@gjcolombo
All zone bundle collections are serialized with respect to each other: instances share a single ZoneBundler that's owned by the InstanceManager, and that has an Arc<Mutex<zone_bundle::Inner>>, which lock is required to collect a bundle.
@bnaecker
Bundle generation probably can be faster, since I did nothing to guarantee their performance. It's copying a bunch of log files from the U.2s into a tarball, which is probably what takes the most time, since there tend to be a lot of files given our log rotation policy. I don't think that itself can be parallelized naively, but I'm sure there's something we can do.
The other thing that may take time is running commands in the zone. It shouldn't take too long, since we run pretty standard things like svcs and netstat -an, but I imagine there's always a way for that to take longer than we want under Tokio.
As for parallelization, I'm not sure. The zone itself needs to be owned while we're taking the bundle (at least, without some invasive changes to the code), to ensure it doesn't disappear. When I wrote this, there was a single lock around the whole sled-agent map for instances, which we also need to take for this to work. I think that stuff may have changed recently, with Sean's work to put the instance runner in a separate task, but I've not looked. There may be opportunities for parallelism now.
Contributor guide
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Trace ZoneBundler ownership through InstanceManager and the zone_bundle::Inner mutex, then inspect bundle collection, log-file tar creation, and commands run in the zone. Compare the sled-agent instance map and instance-runner ownership mentioned in the issue, and define completion as faster concurrent zone-bundle requests that avoid stop-instance timeouts.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- rust
- Domain
- backend, performance
- Issue type
- Refactor
- Difficulty
- 5/5
- Estimated time
- Over a week
- Activity status
- Stale
- Clarity
- Needs clarification
- Newbie friendliness
- 25/100