oxidecomputer / oxidecomputer/propolis
phd-runner is very sensitive to network issues
Nobody has claimed this yet.
- Dominant language
- Rust
- Stars
- 270
- Forks
- 42
- Avg merge
- 4d 5h
- Merged PRs (30d)
- 6
Description
in #941 i saw two failures of phd-run-migrate-from-base, they were something like this:
phd-runner: [TEST - EVENT] Error obtaining artifact from source
e = reqwest::Error { kind: Request, url: "https://oxide-omicron-build.s3.amazonaws.com/alpine.iso", source: hyper_util::client::legacy::Error(Connect, Ssl(Error { code: ErrorCode(5), cause: Some(Io(Os { code: 131, kind: ConnectionReset, message: "Connection reset by peer" })) }, X509VerifyResult { code: 0, error: "ok" })) }
file = phd-tests/framework/src/artifacts/store.rs
line = 627
path = phd_tests::migrate::from_base::migration_from_base_and_back
target = phd_framework::artifacts::store
uri = https://oxide-omicron-build.s3.amazonaws.com/alpine.iso
our approach to downloading then is, per test: try to fetch the artifact once if it is not already accounted for. so in the migration-from-base-and-back case above, we failed the first attempt to get alpine.iso, failed that test, retried on the next test, succeeded, and the rest passed.
it's probably worth trying a second time on a reset, and/or just getting artifacts up front before running tests, or .. something? It's also just not great that we see resets from S3 anyway.
Contributor guide
No contributing guide indexed for this repository
First steps
- Read the whole issue, then the project's contributing guide.
- Comment on the issue to say you are picking it up — it saves two people doing the same work.
- Fork the repository and make your change on a branch.
- Open a pull request that references the issue number.
Research direction
Start at phd-tests/framework/src/artifacts/store.rs lines 623-637 and inspect phd_tests::migrate::from_base::migration_from_base_and_back. Reproduce or observe the S3 artifact fetch failure, then evaluate the retry or prefetch approaches described in the issue; done means transient connection resets no longer cause an unexpected test failure.
Written by the indexing model from the issue text.
Assessment
- Tech stack
- aws, rust
- Domain
- networking, testing
- Issue type
- Bug
- Difficulty
- 4/5
- Estimated time
- 3-5 days
- Activity status
- Stale
- Clarity
- Mostly clear
- Newbie friendliness
- 45/100