oxidecomputer / oxidecomputer/propolis

phd-runner is very sensitive to network issues

Open
#944 0 comments 0 reactions 0 assignees View on GitHub

Nobody has claimed this yet.

testing
Dominant language
Rust
Stars
270
Forks
42
Avg merge
4d 5h
Merged PRs (30d)
6

Description

in #941 i saw two failures of phd-run-migrate-from-base, they were something like this:

	phd-runner: [TEST - EVENT] Error obtaining artifact from source
    e = reqwest::Error { kind: Request, url: "https://oxide-omicron-build.s3.amazonaws.com/alpine.iso", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Io(Os { code: 131, kind: ConnectionReset, message: "Connection reset by peer" })) }, X509VerifyResult { code: 0, error: "ok" })) }
    file = phd-tests/framework/src/artifacts/store.rs
    line = 627
    path = phd_tests::migrate::from_base::migration_from_base_and_back
    target = phd_framework::artifacts::store
    uri = https://oxide-omicron-build.s3.amazonaws.com/alpine.iso

our approach to downloading then is, per test: try to fetch the artifact once if it is not already accounted for. so in the migration-from-base-and-back case above, we failed the first attempt to get alpine.iso, failed that test, retried on the next test, succeeded, and the rest passed.

it's probably worth trying a second time on a reset, and/or just getting artifacts up front before running tests, or .. something? It's also just not great that we see resets from S3 anyway.

Contributor guide

No contributing guide indexed for this repository

First steps

  1. Read the whole issue, then the project's contributing guide.
  2. Comment on the issue to say you are picking it up — it saves two people doing the same work.
  3. Fork the repository and make your change on a branch.
  4. Open a pull request that references the issue number.

Research direction

Start at phd-tests/framework/src/artifacts/store.rs lines 623-637 and inspect phd_tests::migrate::from_base::migration_from_base_and_back. Reproduce or observe the S3 artifact fetch failure, then evaluate the retry or prefetch approaches described in the issue; done means transient connection resets no longer cause an unexpected test failure.

Written by the indexing model from the issue text.

Assessment

Tech stack
aws, rust
Domain
networking, testing
Issue type
Bug
Difficulty
4/5
Estimated time
3-5 days
Activity status
Stale
Clarity
Mostly clear
Newbie friendliness
45/100

Get new issues in your inbox

A short digest of beginner-friendly GitHub issues.